Published 10.14.2026
Speculative Decoding, Multi-Token Prediction, and Chunked Prefill
Summarize with:
TL;DR: Chunked prefill reduces the blocking effect of long prompts, speculative decoding proposes several tokens for a larger model to verify, and multi-token prediction lets a model propose multiple future tokens natively. Each technique addresses a different inference bottleneck, and none guarantees the same speedup across models or workloads.
LLM latency has two major phases. Prefill processes the input prompt and builds attention state. Decode generates output tokens one step at a time. Long-context consumer applications can be slow in either phase, so the right optimization depends on whether the bottleneck appears before or after the first token.
How does chunked prefill improve responsiveness?
Prefill processes the prompt before normal token-by-token generation begins. A long prompt can occupy the accelerator long enough to delay decode work for other users. Chunked prefill divides the prompt into smaller segments so the scheduler can interleave prefill with active decode requests.
The technique can improve fairness and reduce time to first token under mixed workloads, but it introduces scheduling and memory tradeoffs. Chunks that are too large can still block other requests; chunks that are too small can create overhead. The best setting depends on prompt lengths, batch composition, hardware, and the serving engine.
Chunked prefill does not reduce the semantic amount of context the model reads. Prompt compression, retrieval, and memory design address input size; chunking changes how the work is scheduled. Benchmarks should report prompt length, chunk size, concurrency, and whether decode traffic was active during the test.
What is speculative decoding?
Speculative decoding uses a faster draft process to propose several tokens, then asks the target model to verify those proposals in parallel. The accepted tokens advance generation; rejected tokens are replaced using the target model’s distribution. The method can reduce the number of serial target-model steps.
Leviathan, Kalman, and Matias introduced an exact-distribution algorithm at ICML 2023 and reported 2–3× acceleration on T5-XXL with identical outputs. Real-world gains depend on draft speed, acceptance rate, verification efficiency, output length, batch size, and hardware. A poor draft can add work without accepting enough tokens to repay it.
Inworld calls its model-specific implementation realtime decoding. The public explanation should connect that name to the established draft-and-verify technique and disclose where custom attention or verification kernels change the serving path.
How is multi-token prediction different?
Multi-token prediction lets a model or attached heads predict several future tokens from one state. Some models expose a native mechanism that can act as a draft during inference. Unlike a separate small draft model, native MTP can share representations with the target model and avoid maintaining another model.
The serving engine still needs an acceptance and verification strategy. Speed depends on how many useful tokens are proposed, how verification is implemented, and whether additional heads or paths fit the hardware. Quality claims also require evaluation because implementation errors or approximate acceptance can change outputs.
Inworld’s launch material describes a DeepSeek V3.2 replay in which multi-token prediction improved streaming latency from 62.4ms to 39.9ms per token. The result should remain labeled as a replay of the named model and configuration, not a universal DeepSeek or MTP result.
Why do kernel speedups differ from request speedups?
A layer-level microbenchmark isolates one operation, while an end-to-end request includes routing, queueing, prefill, attention, feed-forward layers, sampling, streaming, and network transfer. A 7.5× improvement in verify attention cannot produce a 7.5× request improvement when attention is only part of the total time.
Inworld reports that a split-KV verify kernel made attention 3.4–7.5× faster in Gemma 4 26B layer tests. In a replay of production traffic, the complete realtime-decoding build changed median TTFT from 154ms to 117ms and median inter-token latency from 6.4ms to 5.2ms. The smaller end-to-end gain is the more relevant product result.
| Evidence class | What it can show | What it cannot show alone |
|---|---|---|
| Microbenchmark | Operation-level improvement | Complete request behavior |
| Replay | Effect on recorded prompts | Live capacity and network behavior |
| Production trace | Real workload outcome | Controlled causality |
| Controlled benchmark | Comparable configuration result | Every production workload |
How should quality be protected?
Acceleration can fail silently when precision, draft acceptance, or implementation changes model behavior. Every optimized build should run the same application-specific and model-level evaluations as the baseline. The acceptance method and sampling settings should remain visible.
Inworld reports that an FP8 attention-scaling bug increased performance while reducing Gemma 4 31B’s MATH score; correcting the scaling restored the score from 87.4 to 89.6. A separate configuration reduced serving cost by roughly 80% but lost five MATH points and was not released.
These examples show why speed, cost, and quality belong in one release gate. They remain first-party evidence until the evaluation harness, thresholds, model versions, and raw outputs are published.
When should each technique be used?
| Workload condition | Likely technique | Primary measure |
|---|---|---|
| Long prompts delay active users | Chunked prefill | TTFT under mixed load |
| Decode is serial and draft acceptance is high | Speculative decoding | Accepted tokens per target step |
| Model has a native multi-token path | Multi-token prediction | Streaming latency and quality |
| Repeated long prefixes | Prefix caching plus cache-aware routing | Reused tokens and blended cost |
These methods can be combined, but interactions should be measured. Caching changes prefill work; chunking changes scheduling; speculative decoding changes the decode path. Optimization should begin with a workload trace and end with a full interaction test.
Related Guides
- KV Cache and KV-Aware Routing
- Inference Latency Optimization
- How to Compare Inference Providers
- Consumer Inference Cloud
- How to Host Open Models in Production
Key Takeaways
- Chunked prefill schedules long prompts in smaller units so active decode requests are not blocked as long.
- Speculative decoding can preserve the target distribution while reducing serial target-model steps.
- Multi-token prediction is a native draft path whose value depends on acceptance and verification efficiency.
- Kernel-level speedups are not equivalent to end-to-end request improvements.
- Every optimized build needs quality, latency, cost, and reliability gates.
Frequently Asked Questions
Does speculative decoding always produce identical output?
Exact speculative-decoding algorithms preserve the target model’s distribution, but individual sampled text can still vary when sampling is stochastic. Approximate implementations, configuration changes, or bugs can alter behavior. Providers should disclose the acceptance method and show quality results alongside speed measurements.
Can chunked prefill reduce prompt cost?
Chunking changes scheduling rather than the number of input tokens, so it does not inherently reduce billed prompt cost. It can improve responsiveness and utilization. Prefix caching, prompt compression, retrieval, or shorter memory can reduce repeated input work and may lower cost depending on provider billing.
What is draft acceptance rate?
Draft acceptance rate measures how often tokens proposed by the draft path are accepted by the target model. Higher acceptance usually allows more progress per verification step, but the result also depends on draft cost and kernel efficiency. Report accepted tokens per target step with end-to-end latency.