warming up your workspace

How speculative decoding makes LLMs faster without changing a single output

Large language models generate text one token at a time, and each token costs a full forward pass through the whole network. That serial bottleneck is why a big model feels slow: it cannot start token two until token one is done. Speculative decoding is the trick that broke this open, and by 2026 it runs inside every serious inference engine. It makes generation two to three times faster, and the beautiful part is that it changes nothing about the output. The text you get is drawn from exactly the same distribution as before. That sounds impossible. Build the acceptance rule and you see why it is not.

The one idea

Keep two models: a big accurate one (the target) and a tiny fast one (the draft). Let the cheap draft model quickly guess the next several tokens. Then run the expensive target model once over that whole guessed sequence, which it can do in a single pass because verifying several tokens in parallel costs about the same as generating one. For each guessed token, a clever accept-or-reject rule decides whether to keep it. The tokens the draft got right are accepted for free; the first one it gets wrong is corrected and the rest thrown away. Most of the time the draft is right about the easy tokens, "the", "of", the end of a common word, so you get several tokens per expensive pass instead of one.

The magic is the acceptance rule, which is designed so the tokens that survive are distributed identically to sampling from the target directly. You pay less and get the same thing.

Build the acceptance rule

For each drafted token t, accept it with probability min(1, target(t) / draft(t)). If the draft is confident about a token the target also likes, that ratio is near one and it sails through. If the draft over-weighted a token, it sometimes gets rejected, and on rejection you resample from the "residual" distribution, the leftover probability the target wanted but the draft did not cover, then stop.

import numpy as np
rng = np.random.default_rng(7)

def speculative_step(target, draft, K):
    accepted = []
    for _ in range(K):
        t = sample(draft)                              # draft's cheap guess
        if rng.random() < min(1.0, target[t] / draft[t]):
            accepted.append(t)                         # target agrees, keep it
        else:
            resid = np.maximum(target - draft, 0)      # what target wanted extra
            resid = resid / resid.sum()
            accepted.append(sample(resid))             # corrected token
            break                                      # stop at first miss
    return accepted

Three details that matter:

  • The min(1, target/draft) cutoff plus the residual resampling is not a heuristic, it is a proof. Together they guarantee the accepted token has exactly the target's probability. This is the same math as rejection sampling, repurposed so a wrong guess is never simply discarded but replaced by a correctly distributed token.
  • You stop at the first rejection. Everything the draft proposed after a miss is based on a token that did not survive, so it is invalid and dropped. That is why the speedup depends on how often the draft agrees: agreement lets you keep a long run.
  • The draft must be cheap relative to the target, otherwise you spend more on guessing than you save on verifying. A model ten times smaller, or a few extra prediction heads bolted onto the target itself, is the usual choice.

Proof: faster, and provably the same output

Run the draft proposing up to four tokens per target pass, then check the output distribution against plain target sampling:

draft proposes up to 4 tokens per pass
avg tokens accepted per target pass: 3.66
so ~3.66x fewer expensive target calls

target dist : [0.4  0.25 0.15 0.1  0.06 0.04]
speculative : [0.401 0.25  0.15  0.1   0.06  0.04]
max abs diff: 0.0006

Two claims, both confirmed. The speedup is real: 3.66 tokens came out per expensive target pass instead of one. And the output is unchanged: the tokens speculative decoding produces match the target model's own distribution to within sampling noise. You did not trade quality for speed. You got the speed for free, by letting a fast model do the easy guessing and reserving the slow model for checking.

Where this shows up

This is standard in production LLM serving, vLLM, TensorRT-LLM, and the rest ship it, often with self-speculation variants like Medusa and EAGLE that avoid a separate draft model by giving the target extra lightweight heads that predict a few tokens ahead. The acceleration is largest exactly where language is predictable, boilerplate, code, structured output, which is much of what these models generate. It is one of the few optimizations in machine learning that is genuinely lossless: same samples, less compute.

If you want to build the sampler, the model these tokens come from, and the serving loop around it, that is the path the ai track on IWTLP builds from scratch.

Sources