How a tokenizer works, by building byte-pair encoding

A language model never sees your words. It sees tokens: integer IDs for chunks of text a tokenizer carved out first. Those chunks are why "strawberry" might be three tokens, why code costs more than prose, and why your API bill is measured in tokens, not characters. The algorithm behind most of them, byte-pair encoding, is short enough to build.

The one idea

A tokenizer needs a fixed vocabulary, but text is open-ended (new words, typos, code, emoji). BPE solves this by starting from single characters and greedily merging the most frequent adjacent pair into a new symbol, over and over. Common sequences like th or the become single tokens; rare words fall back to smaller pieces. The vocabulary stays finite, and nothing is ever unrepresentable.

Count the pairs

BPE trains on a corpus. First, split words into characters (with a marker so we know where words end), and count how often each adjacent pair occurs.

from collections import Counter

def get_pairs(words):
    pairs = Counter()
    for word, freq in words.items():
        symbols = word.split()
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs

# corpus as {word-with-spaces-between-chars: frequency}
words = {"l o w </w>": 5, "l o w e r </w>": 2,
         "n e w e s t </w>": 6, "w i d e s t </w>": 3}
print(get_pairs(words).most_common(1))   # [(('e', 's'), 9)]

The most frequent pair is ('e', 's'), appearing 9 times across newest and widest. That is our first merge.

Merge, and repeat

Merging means replacing every occurrence of the pair with a joined symbol, then recounting, because new pairs appear once old ones fuse.

def merge(pair, words):
    a, b = pair
    joined = a + b
    out = {}
    for word, freq in words.items():
        out[word.replace(f"{a} {b}", joined)] = freq
    return out

def train(words, num_merges):
    merges = []
    for _ in range(num_merges):
        pairs = get_pairs(words)
        if not pairs:
            break
        best = pairs.most_common(1)[0][0]
        words = merge(best, words)
        merges.append(best)
    return merges, words

merges, final = train(words, 8)
print(merges)

[('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ...]

Watch what happened: e+s became es, then es+t became est, then est+</w> became a single end-of-word token est</w>. BPE discovered the suffix "est" because it kept recurring. Nobody told it about suffixes; frequency did.

Tokenize new text with the learned merges

To encode a new word, apply the merges in the order they were learned. Anything not merged stays as small pieces, so an unseen word never breaks the tokenizer.

def encode(word, merges):
    symbols = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(symbols) - 1:
            if symbols[i] == a and symbols[i + 1] == b:
                symbols[i:i + 2] = [a + b]
            else:
                i += 1
    return symbols

print(encode("lowest", merges))   # ['low', 'est</w>']
print(encode("zqx", merges))      # ['z', 'q', 'x', '</w>']

lowest split into just two tokens (low + est</w>), because both were learned from the corpus. The nonsense zqx fell back to characters, still fully representable. That fallback is the whole reason BPE beats a fixed word list: it degrades gracefully.

Why token count is your bill

Now the practical payoffs are obvious. Common English is dense, whole words in one or two tokens, so prose is cheap. Rare words, unusual names, and especially code (__init__, deep indentation, symbols) fragment into many tokens, so they cost more per character. A model's context window is measured in tokens, so "how much can it read" depends on what you feed it. And "strawberry" being split oddly is why models sometimes miscount its letters: they never saw the letters, only the chunks.

Understanding the tokenizer turns those quirks from surprises into predictions. It is the first stage of every NLP pipeline, and the first thing you build in the NLP track, because everything downstream, embeddings, attention, generation, operates on the tokens this step produces.