Get the 2026 ML Training Cookbook | 52 recipes — GRPO, Flow Matching, World Models, and everything in between Download Now →
Tokenization
The process of breaking text into discrete units (tokens) that a model can process, bridging raw text and numerical representations.
Tokenization is the process of breaking input text into discrete units called tokens, which serve as the fundamental input units for language models. Since neural networks operate on numerical data, text must be converted into a sequence of integer token IDs before processing. The tokenization scheme determines how text is segmented — into individual characters, whole words, or subword units — and directly affects the model's vocabulary size, sequence length, and ability to handle rare or unseen words.
Modern language models predominantly use subword tokenization algorithms such as Byte-Pair Encoding (BPE), WordPiece, or SentencePiece. These algorithms learn a vocabulary of frequently occurring character sequences, balancing between character-level tokenization (which produces long sequences but handles any text) and word-level tokenization (which produces short sequences but has a large, open-ended vocabulary and cannot handle out-of-vocabulary words). Subword tokenization splits rare words into known subword units (e.g., 'unbelievable' → 'un', 'believ', 'able'), ensuring any text can be tokenized.
The choice of tokenizer has significant practical implications. It determines the number of tokens per input (which affects compute cost, since attention is quadratic in sequence length), the model's multilingual capability (tokenizers trained primarily on English text may produce very long token sequences for other languages), and how the model handles numbers, code, and special characters. Tokenizer quality is a frequently overlooked factor in model performance.
- ■Converts raw text into integer token IDs for neural network processing
- ■Modern models use subword algorithms: BPE, WordPiece, SentencePiece
- ■Subword tokenization handles rare/unseen words by splitting into known units
- ■Affects sequence length, compute cost, and multilingual capability
- ■Tokenizer quality is an often-overlooked factor in model performance
Using a BPE tokenizer with a 50,000-token vocabulary to process multilingual customer support tickets, where rare words are split into subword units rather than mapped to an unknown token.