AI GlossaryTokenizationENTRY — Natural Language Processing
[Natural Language Processing]
Tokenization.
Breaking text into smaller units (tokens) such as words, subwords, or characters.
In-depth explanation
01Tokenization is the first step in NLP pipelines, converting raw text into a sequence of tokens that models can process. Word tokenization splits on spaces; subword tokenization (like BPE, WordPiece) handles unknown words better. Each token is then typically mapped to a numerical ID. Tokenization choices significantly impact model performance.
Examples
02EX. 01
"Hello world" → ["Hello", "world"]
EX. 02
Subword: "unhappiness" → ["un", "happiness"]
More in Natural Language Processing
0301BERTBidirectional Encoder Representations from Transformers, a pre-trained language model for NLP tasks.02Named Entity Recognition (NER)Identifying and classifying named entities in text into categories like person, organization, location.03Natural Language Processing (NLP)The field of AI focused on enabling computers to understand, interpret, and generate human language.04Sentiment AnalysisDetermining the emotional tone or opinion expressed in text, typically positive, negative, or neutral.05Word EmbeddingDense vector representations of words that capture semantic meaning and relationships.
[NEXT] — APPLY THE CONCEPT
Master Tokenization.
Learn how to apply this concept with hands-on projects in our comprehensive AI programs.