N-gram language models and perplexity, the model GPT made obsolete but not irrelevant

Before neural networks took over language, a language model was something you could read: a giant table of counts saying "after the words the cat, the next word was sat this often, ran that often". These are n-gram models, and while a transformer will crush them on any real task, they are the clearest possible way to learn what a language model is and how we grade one. And the grade, perplexity, is not a historical footnote. It is the exact metric printed in the evaluation section of every large-model paper in 2026. Build the model and the metric together and both stop being jargon.

The one idea

A language model assigns a probability to the next word given the words before it. An n-gram model makes one brutal simplification: only the last n minus 1 words matter. A bigram model (n = 2) looks at just the previous word; a trigram (n = 3) at the previous two. You estimate the probabilities by counting: to find the chance of sat after the cat, count how often the cat sat appeared and divide by how often the cat appeared. That is the entire model, a dictionary of contexts, each mapping to a distribution over next words. No training loop, no gradients. Just counting.

Build it

Count the contexts, then turn counts into smoothed probabilities.

from collections import defaultdict, Counter

def train_ngram(tokens, n):
    ctx = defaultdict(Counter)
    for i in range(len(tokens) - n + 1):
        prefix = tuple(tokens[i:i+n-1])          # the n-1 words of context
        ctx[prefix][tokens[i+n-1]] += 1          # count what followed
    return ctx

def prob(ctx, prefix, word, vocab, k=1):         # add-k smoothing
    counts = ctx[prefix]
    return (counts[word] + k) / (sum(counts.values()) + k*vocab)

Three details that matter:

• The +k in prob is smoothing, and it is not optional. Without it, any word the model never saw after a given context gets probability zero, and a single zero makes the probability of a whole sentence zero, which breaks the math. Add-k pretends every word was seen a fraction of a time, so nothing is impossible, just unlikely.
• Higher n captures more context but splinters your data. There are far more distinct three-word contexts than two-word ones, so each is seen fewer times, and the counts get unreliable. This is the sparsity problem, and it is the ceiling n-grams cannot break through.
• The model is just a nested dictionary. You can print it and read it. That transparency is exactly what neural language models traded away for power.

Measure it with perplexity

Perplexity asks: on held-out text the model did not train on, how surprised is it? Formally it is two raised to the average number of bits the model needs to encode each word. Low perplexity means the model consistently put high probability on the words that actually came next. The intuition: perplexity is roughly "how many words was the model effectively choosing between at each step". A perplexity of 4 means it was about as uncertain as a fair 4-sided die.

def perplexity(ctx, tokens, n, vocab):
    logp = 0.0; N = 0
    for i in range(len(tokens) - n + 1):
        prefix = tuple(tokens[i:i+n-1])
        p = prob(ctx, prefix, tokens[i+n-1], vocab)
        logp += math.log2(p); N += 1
    return 2 ** (-logp / N)

Proof: more context helps, until the data runs out

Train unigram, bigram, and trigram models on a small corpus and score the same test sentence:

1-gram perplexity on test: 9.10
2-gram perplexity on test: 4.63
3-gram perplexity on test: 5.24

The story is in those three numbers. Going from unigram to bigram nearly halves perplexity, from 9.1 to 4.6, because knowing the previous word tells you a lot about the next one. But the trigram is worse than the bigram here, not better. That is not a bug; it is the sparsity ceiling made visible. On this tiny corpus, most two-word contexts were seen only once or twice, so the trigram's counts are dominated by the smoothing term rather than real evidence. More context needs more data, and when you do not have it, the extra context hurts. Meanwhile the model clearly learned real structure, greedy generation from the cat produces the cat sat on the mat . the dog sat, lifted straight from the patterns it counted.

Where this shows up

Perplexity is the throughline from these count tables to today's models. When a 2026 language model reports its quality on a benchmark, perplexity is usually the number, same definition, computed the same way, just with a neural network producing the probabilities instead of a dictionary of counts. And the sparsity problem you just watched is why neural models won: they generalize across contexts, so the cat sat can inform the dog sat even if the exact phrase was never seen, something a count table can never do. N-grams still live inside spell checkers, input prediction, and speech recognizers where their speed and transparency earn their keep.

If you want to build from n-grams up through word embeddings to the attention that replaced them, that is the arc the nlp track on IWTLP walks, each model measured by the perplexity you just implemented.