Speculative Decoding: Why Drafting Tokens Can Be Free
Speculative decoding can make drafting tokens effectively free because a small, fast model proposes several tokens while a larger model checks them together in one parallel pass. The draft computation still uses resources; the speedup comes when verification costs little more than producing one large-model token, and accepted proposals avoid repeated, sequential large-model steps.
What “free” means in speculative decoding
“Free” is shorthand for a particular comparison: the extra draft tokens may add little or no latency to the large model’s verification step. It does not mean computation, electricity, memory, or hardware are free. A draft model must run, and the target model must evaluate the proposal. Whether the method saves time or money depends on how those costs compare with ordinary generation.
Autoregressive language models ordinarily generate one token at a time. To predict token two, the model needs the result for token one; to predict token three, it needs token two, and so on. That dependency makes generation sequential, even when a model can process a large prompt in parallel. Speculative decoding tries to reduce the number of sequential steps performed by the expensive target model.
How drafting and verification work
A fast draft model proposes a short continuation. The target model then evaluates the proposed tokens in parallel, using a causal attention mask so each position is scored with only the preceding context. This is similar to processing a short sequence in one forward pass rather than asking the target model to generate every token in separate passes.
- Draft: The smaller model proposes a sequence, such as “the answer is probably 42.”
- Verify: The target model calculates the next-token distributions for the proposed positions in a batched pass.
- Accept or correct: Proposed tokens are accepted only when the verification rule permits them. If a proposal fails, generation continues from the appropriate corrected token.
- Repeat: The process drafts and verifies another segment until generation ends.
Because modern accelerators handle parallel work efficiently, scoring several positions can take far less time than generating the same number of tokens serially. The target model still does real work for the full proposed segment, but much of that work happens concurrently. If the hardware has spare parallel capacity, the added cost of checking a few more positions can be small.
Why the target model’s output can remain exact
Correct speculative decoding is not simply “trust the small model when it seems right.” In standard sampling-based methods, an acceptance-and-correction procedure is designed so the final output distribution matches the target model’s distribution. The verifier compares the draft model’s proposal with the target model’s probabilities; when a token is rejected, a correction is sampled according to the method’s rules.
For greedy decoding, the verifier can accept a draft token when it agrees with the target model’s next-token choice. Otherwise, it uses the target model’s choice at the first disagreement. For probabilistic sampling, maintaining the target distribution requires a more careful acceptance rule than merely accepting every plausible token. Exactness claims depend on using the proper algorithm and matching assumptions, including how the models handle the same context and tokenization.
This distinction matters: a system that just substitutes a smaller model’s answer can change output quality or probabilities. Speculative decoding is useful precisely because its verification step can preserve the target model’s behavior while taking advantage of good proposals.
What determines the speedup?
The main factor is how many draft tokens the target model accepts on average. If drafts often match what the target would produce, one verification pass advances generation by several tokens. If they often fail immediately, the method pays for drafting and verification but gets little sequential work done in return.
- Draft quality: A draft model that predicts the target’s likely continuation well tends to achieve longer accepted runs.
- Draft cost: A larger or slower draft model can eat into the time saved, even if it proposes good tokens.
- Verification efficiency: Batch size, accelerator utilization, memory traffic, and implementation affect how costly parallel checks are.
- Workload: Short outputs, long prompts, and different decoding settings can change the balance between setup, drafting, and verification.
- Draft length: Longer proposals may offer more potential parallelism, but also increase the chance that verification rejects a token early.
There is no universal speedup. An implementation should measure end-to-end latency or throughput on its actual model, hardware, and workload. A faster token-generation loop does not automatically mean lower total cost if it requires additional model memory or specialized infrastructure.
Why “free” does not mean zero cost
The target model’s parallel verification is not costless: it consumes compute and memory bandwidth. Drafting also adds inference work, and both models may need to remain available in memory. In some settings the draft model is small enough that this overhead is outweighed by fewer target-model decoding steps; in others, the target model is already efficient enough that speculation brings little benefit.
It is helpful to distinguish three meanings of cost:
- Latency: Extra proposed tokens can be nearly free in elapsed time if they fit efficiently into a verification pass.
- Compute: The system still performs draft and verification operations, so total computation is not zero.
- Serving expense: Actual cost depends on hardware utilization, memory requirements, deployment, and how many requests the system can handle.
Thus, “free drafting tokens” describes marginal cost under favorable conditions, not a guarantee that every speculative token has no resource cost.
Common limitations and misconceptions
Speculation works best when the draft model and target model tend to agree. A mismatch in tokenization or a difficult continuation may reduce acceptance. Sampling settings can also affect agreement: highly varied outputs are harder for a draft model to predict than a deterministic continuation. Systems can adapt the draft length, but tuning is a trade-off rather than a way to eliminate all overhead.
Another misconception is that the target model checks only whether a draft sentence “makes sense.” It checks token-level probabilities or choices according to the decoding method. Nor does speculative decoding necessarily make the model’s answer better; its primary goal is to produce the target model’s output faster. Quality changes would indicate a different algorithm or implementation behavior, not an inherent benefit of speculation.
Frequently Asked Questions
Are speculative decoding tokens actually free?
Not literally. Drafting and verification consume compute. They can be nearly free in latency when the target model checks several proposed tokens in a pass that costs little more than a single sequential step.
Does speculative decoding change the answer?
A correctly implemented method can preserve the target model’s output distribution. Simply using a smaller model’s proposals without the required verification and correction can change it.
Why use a small model to draft?
A smaller model can propose tokens quickly, creating a candidate continuation for the larger model to verify. The approach is worthwhile when the draft’s speed and agreement offset its extra work.
Does it always make text generation faster?
No. Low acceptance, expensive drafting, short outputs, or inefficient verification can erase the advantage. Performance depends on the model pair, hardware, and workload.
Is speculative decoding the same as batching?
No. Batching processes multiple requests or examples together. Speculative decoding proposes future tokens for one sequence and verifies them in parallel, though implementations can also use batching.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.