Speculative Decoding: How Guessing Tokens You Then Throw Away Speeds Up LLM Inference

Here’s a claim that sounds like it can’t possibly be true: you can use a small, weak model to guess several of a large model’s next tokens ahead of time, throw away every guess the large model disagrees with, and end up generating text exactly as if the small model had never been involved at all — same distribution, same quality, zero approximation — while going meaningfully faster. Not “faster with a small quality hit you probably won’t notice.” Exactly the same output distribution. Faster is the whole trade, with no other side to it.

That claim is speculative decoding, and the fact that it sounds like a scam is precisely why the mechanism behind it is worth understanding in full rather than taking on faith.

The bottleneck this is actually exploiting

If you’ve read vLLM Internals, you already know the load-bearing fact this post builds on: a single decode step — producing one token, for one sequence, autoregressively — is memory-bandwidth bound, not compute bound. Every decode step has to stream the entire model’s weights (tens of gigabytes for anything modern-sized) plus the growing KV cache out of HBM and through the SMs, to do a comparatively tiny amount of arithmetic: produce one token’s worth of logits. The GPU’s compute units spend most of that step idle, waiting on memory traffic.

Continuous batching fixes this one way: put more sequences in flight so that one expensive weight-read gets amortized across many tokens produced simultaneously, one per sequence. But that only helps when you have many concurrent sequences to batch. A single interactive session — one user, one conversation, decoding one token at a time — can’t be batched with itself. Each token still costs a full weight-read.

Speculative decoding attacks the same underlying waste from a different angle: instead of finding other sequences to batch with, it tries to extract more than one token’s worth of useful work out of a single expensive weight-read of the big model, for the same sequence.

The mechanism: propose cheap, verify expensive, in parallel

The setup needs two models:

• A target model — the big model whose output distribution you actually want (say, a 70B-parameter model).
• A draft model — a much smaller, much cheaper model that’s a reasonable approximation of the target (say, a 1B-parameter model from the same family, or a distilled variant).

Each round works like this:

    sequenceDiagram
	    participant D as Draft model (small)
	    participant T as Target model (big)
	    participant V as Verify + accept/reject
	
	    D->>D: sample token 1 (autoregressive)
	    D->>D: sample token 2 (autoregressive)
	    D->>D: ... sample token γ (autoregressive)
	    D->>T: propose γ draft tokens
	    T->>T: ONE forward pass over all γ+1 positions (parallel, like a prefill)
	    T->>V: target-model probabilities at each position
	    V->>V: walk positions left→right, accept/reject each draft token
	    V->>V: on first rejection: resample a correction token, stop
	    V->>V: if all γ accepted: sample one bonus token from target (free)

The draft model runs autoregressively for γ (gamma) steps to propose γ tokens — cheap, because it’s small. Then the target model evaluates all γ+1 positions in a single forward pass, the same way it would evaluate a prompt during prefill, rather than as γ separate sequential decode steps. That’s the trick: verifying γ tokens in parallel costs roughly the same one weight-read as producing a single token normally would, instead of γ weight-reads.

Now the actual hard part: how do you decide which draft tokens to keep, without biasing the output toward whatever the small model happens to prefer?

The part that makes it exact, not approximate

The naive approach — “accept a draft token if the target model’s greedy argmax agrees with it” — would be wrong. It would silently skew the output distribution toward the draft model’s preferences whenever the two models disagree about which correct-ish token to pick, even in cases where the target model would have been happy to sample either one.

Both papers that introduced this technique in essentially the same form — Leviathan et al., Fast Inference from Transformers via Speculative Decoding (Google Research), and Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (DeepMind), both circa late 2022/early 2023 — use modified rejection sampling instead. For each speculated position, with the draft model’s probability q(x) for the token it actually sampled and the target model’s probability p(x) for that same token at that same position:

def speculative_accept(p_target, q_draft, token_id, rng):
    # p_target, q_draft: full probability distributions at this position
    # token_id: the specific token the draft model sampled
    accept_prob = min(1.0, p_target[token_id] / q_draft[token_id])
    if rng.random() < accept_prob:
        return token_id, None          # accepted — draft token stands

    # rejected: don't just fall back to the target's argmax.
    # Sample a correction from the *residual* distribution instead.
    residual = torch.clamp(p_target - q_draft, min=0.0)
    residual = residual / residual.sum()
    correction_token = torch.multinomial(residual, 1).item()
    return None, correction_token

Walk the γ speculated positions left to right, applying this per-token. The moment one is rejected, stop — discard every speculated token after it (they were conditioned on a token that never actually happened in the final output), sample the correction token from the residual, and end the round there. If every single one of the γ tokens gets accepted, you additionally get to sample one more token for free from the target model’s own distribution at the new position — free because you already computed that distribution during the parallel verification pass.

The residual step — sampling from max(0, p - q), renormalized — is what makes this exact rather than approximate. Intuitively: whenever the draft model over-proposes a token relative to how much probability mass the target model actually assigns it (q(x) > p(x)), rejecting it and sampling from the leftover mass p - q gives back exactly the probability the draft “stole.” Work through the algebra over the full accept/reject/resample process and the marginal distribution over the final sampled token is provably identical to just sampling from the target model directly. This is the single fact worth internalizing about this whole technique: speculative decoding is not a quality/speed trade-off. It’s a latency optimization that happens to leave the output distribution completely untouched. The draft model never gets to decide what the target model “would have said” — it only ever gets to propose, and the target model’s own math is what actually arbitrates every accepted token.

How much speedup you actually get

Let α (alpha) be the expected per-token acceptance rate — how often, on average, a draft-proposed token survives the accept/reject step — and γ the number of tokens speculated per round. The expected number of tokens produced per round (one draft pass plus one verify pass) works out to:

E[tokens per round] = (1 − α^(γ+1)) / (1 − α)

At α = 0.9 and γ = 4, that’s (1 − 0.9⁵)/(1 − 0.9) ≈ 4.1 tokens per round instead of 1 — call it roughly a 4x reduction in the number of target-model weight-reads needed to produce that much text. At α = 0.5 and the same γ = 4, it drops to about 1.9 tokens per round — still a win, but a much smaller one, because more rounds end in an early rejection.

That formula only counts target-model cost, though, and the draft model isn’t free — it runs γ times per round. The expected wall-clock speedup has to be discounted by roughly 1 / (1 + γ·c), where c is the ratio of one draft-model forward pass’s cost to one target-model forward pass’s cost. This is why draft model choice matters more than draft model speed past a certain point: a draft model that’s extremely cheap (c near 0) but has a low acceptance rate (α low, because it’s a poor approximation of the target) can end up slower in wall-clock terms than a slightly more expensive draft model that agrees with the target far more often. You’re not just buying cheap tokens — you’re buying cheap tokens the big model is likely to actually keep.

This is also why acceptance rate is domain-sensitive in a way that catches people off guard: a small general-purpose chat model might have a high α on conversational text and a much lower one on, say, dense code with unusual variable names — because that’s exactly the kind of input where a much smaller model’s next-token distribution diverges hardest from a much larger one’s.

Draft models aren’t the only way to propose

Training and hosting a separate draft model is real overhead — extra GPU memory, an extra model to keep in sync as the target model gets updated, extra serving-loop complexity. Production serving engines, vLLM included, support alternatives that sidestep hosting a second model entirely:

• N-gram / prompt-lookup decoding. Instead of a neural draft model, propose the next few tokens by searching for the current token sequence’s recent n-gram elsewhere in the prompt or generated output so far, and speculating whatever followed it last time. This costs almost nothing and works surprisingly well on tasks with a lot of verbatim repetition or copying — code editing, summarization of a document the model is quoting from, structured data transformation — precisely the cases where the “draft” doesn’t need any real modeling capability, just pattern matching against text that’s already in context.
• Self-speculative / integrated heads — approaches like Medusa and EAGLE attach extra lightweight prediction heads directly onto the target model, trained to predict a few tokens ahead using the target model’s own hidden states, rather than running a fully separate model. This avoids hosting two models altogether, at the cost of extra training work up front and extra heads baked into the deployed model.

All of these plug into the exact same accept/reject verification machinery described above — the correctness guarantee doesn’t care where the draft tokens came from, only that you have some q(x) (or a stand-in for one) to weigh against the target’s p(x).

Where this breaks down

• It’s a wall-clock optimization, not a throughput-under-load one. Speculative decoding buys you fewer target-model weight-reads per token for a single sequence’s latency. Under heavy concurrent load, where continuous batching already keeps the GPU compute-bound rather than memory-bound, the marginal benefit shrinks — you’re optimizing a bottleneck that’s less binding once the batch is already large. It’s most valuable for low-concurrency, latency-sensitive serving (interactive single-user sessions), less so for maximizing aggregate tokens/sec on a saturated multi-tenant server.
• A poorly matched draft model can make things slower, not faster. Low acceptance rate plus non-trivial draft cost means you pay for γ extra forward passes per round and get little of the parallel-verification benefit back. This isn’t a tunable knob you can ignore — picking or training a draft model that actually tracks the target’s distribution on your real traffic matters.
• It costs real serving-loop complexity. Batching draft-then-verify rounds across many concurrent requests, each accepting a different number of tokens per round, interacts non-trivially with continuous batching’s scheduler and with KV cache management (rejected speculative positions’ KV entries have to be discarded, not just the tokens). This is systems engineering, not a drop-in flag with no downstream effects on the rest of the serving stack.
• It needs a draft model (or n-gram source, or extra heads) to exist at all. For a brand-new target model with no smaller sibling in the same family and no obvious n-gram-friendly workload, there may simply be no good proposal source available yet.

The takeaway

The headline fact is worth restating on its own: this is one of a small number of LLM inference optimizations that costs you nothing in output quality — not “negligible,” not “within noise,” but provably, exactly nothing, by construction of the rejection-sampling step. Nearly every other lever in this space (quantization, distillation, early-exiting) trades some accuracy for speed and asks you to decide how much you’re willing to give up. Speculative decoding doesn’t ask that question, because the answer is always zero. The entire engineering problem it presents is instead about acceptance rate and draft cost — finding or building a cheap-enough proposer that the big model agrees with often enough, for your actual traffic, to make the extra forward passes worth it.