Last updated: July 7, 2026
Inference latency optimization is the work of cutting the delay between a user's request and the model's response. Inworld AI, a research lab and inference provider focused on realtime consumer AI, runs speculative decoding, KV caching, quantization, and custom kernels in production; speculative decoding alone speeds decode 2 to 3x with provably identical output (
Leviathan et al., 2022).
This page defines the metrics, maps each lever's gain against its cost, and shows how to measure your own TTFT in ten lines.
What metrics actually measure inference latency?
Four numbers describe inference latency: time-to-first-token (TTFT), the delay before the first token streams back; inter-token latency (ITL), the gap between subsequent tokens; time-to-first-audio (TTFA) for voice pipelines; and the percentile you read them at. Median (P50) describes the typical request; P90 and P99 describe the requests users complain about.
TTFT is dominated by prefill compute, queueing, and network round trip; ITL by decode speed and batch contention, so they respond to different levers. In a voice pipeline the budget compounds: TTFA stacks the LLM's TTFT on top of the speech model's latency. Inworld's Realtime TTS-2 delivers sub-250ms P90 time-to-first-audio and TTS 1.5 Mini roughly 100ms median (
inworld.ai/tts, as of July 7, 2026), leaving most of a conversational turn's budget to the LLM.
Which levers reduce latency, and what does each cost?
Six serving-side levers dominate the optimization stack: speculative decoding, prefix caching, quantization, batching-window tuning, disaggregated prefill/decode, and kernel optimization. Each buys a specific improvement and costs something specific, whether memory, throughput, marginal accuracy, or engineering time. None changes model outputs except quantization.
Prefix caching pays off disproportionately in chat products because sessions share long system prompts and grow multi-turn history, so cache hit rate directly cuts both TTFT and cost (
vLLM automatic prefix caching). Speculative decoding works best exactly where consumer dialogue lives: predictable, stable-persona text where the draft model guesses well. Beneath both sits continuous batching, which keeps the GPU saturated by admitting and evicting requests at each decoding step, an order-of-magnitude throughput gain over naive serving (
Anyscale); batching-window tuning trades a slice of that throughput back for latency. Disaggregation, separating compute-bound prefill from bandwidth-bound decode onto dedicated pools (
DistServe), is an architecture decision, not a config flag.
Measured: the same open models on different hosts
The levers above are abstract until you point a stopwatch at them, so on July 7, 2026 we did: K=5 serial streamed chat completions per model-host cell, sent from a single client, all through the same Router endpoint, so the network path is identical for every row. TTFT is measured from request start to the first content token. One asymmetry to state plainly: third-party hosts include the routing hop, while Inworld-served models are colocated with the endpoint; that hop is part of what a Router user actually experiences when calling either. These are selected results; the full raw data is reproducible with the method described here and the script below.
All figures measured July 7, 2026. The two reference rows are a different model class entirely, frontier proprietary APIs rather than served open weights, and are included for orientation only. On Gemma 4 31B, DeepInfra's deployment did not produce stable results in our runs (2 of 5 requests failed), so we exclude its numbers rather than report them. Single-day, single-region numbers move; treat this table as a snapshot of what the levers above produce in practice, and measure your own workload before deciding.
GLM-5.1 and GLM-5.2 are reported separately because both are reasoning models: they stream reasoning deltas before the first content token, so for these cells we measured time to first delta of any kind (TTFD) with max_tokens raised to 2,000. Same method and day otherwise: GLM-5.1 on Inworld reached 692 ms median TTFD at 38 tokens per second, against 890 ms and 36 tokens per second for the same model on DeepInfra. GLM-5.2 split, with DeepInfra faster to first delta (727 ms vs 904 ms) and the Inworld deployment sustaining higher throughput (22 vs 14 tokens per second). The measurement lesson doubles as integration advice: benchmark reasoning models to first delta and budget generous max_tokens, or the model will look broken when it is merely thinking.
Why are the biggest wins architectural?
Latency is the product of three choices: which model you run, what serving stack runs it, and where it runs relative to your users. In practice a smaller or quantized model on an optimized stack routinely beats a larger model on generic serving, at lower cost per token.
Work the three axes in order:
- Model choice. The easiest millisecond to save is the one a smaller model never spends. If a 26B or 31B open model clears your quality bar, its prefill and decode are structurally faster than a frontier model's, and it costs less to serve. Inworld serves Gemma 4 26B A4B and Gemma 4 31B first-party; current per-token rates are on inworld.ai/models.
- Serving stack. The same weights on the same GPU vary enormously by engine. Inworld's published example: a custom FlashInfer kernel patch (flashinfer-ai PR #2959) took Gemma 4 31B Dense to roughly 27K tokens/sec at p50 TTFT of 1.7s under saturation load, about 4x pre-patch throughput on identical hardware (hosting guide, May 2026). A 4x serving gap swamps most parameter tuning.
- Network placement. Round-trip time adds to TTFT before any GPU does anything; a cross-continent hop can cost more than a well-served model's entire prefill. Regional placement next to your app backend and TLS connection reuse are levers of the same magnitude as the serving techniques above. GPU price also varies sharply by cloud and region; the GPU cloud comparison shows April 2026 B200 list rates spanning $3.49 to $14.24 per GPU-hour.
How do you measure TTFT yourself?
Measure from your own backend region, not from a vendor's benchmark page. Send a streaming request, record the time until the first content chunk arrives, repeat 20 or more times, and read the median and the worst case. curl's %{time_starttransfer} or a ten-line Python script is enough to compare providers honestly.
for i in 1 2 3 4 5; do
curl -s -o /dev/null \
-w "run $i TTFT (time to first byte): %{time_starttransfer}s\n" \
-X POST https://api.inworld.ai/v1/chat/completions \
-H "Authorization: Basic $INWORLD_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "inworld/models/gemma-4-31b-it", "messages": [{"role": "user", "content": "Say hello in one sentence."}], "stream": true}'
done
Three caveats keep the numbers honest. Discard the first run: it pays TLS setup that a warm production client will not. First-byte timing is a close proxy for TTFT on streaming endpoints but includes network round trip, which is the point if you are measuring what users feel. And run the test at your traffic peak; queueing under load is where P99 diverges from the demo number.
When is latency optimization premature?
When no human is waiting on the response. Batch workloads, overnight enrichment and evaluation jobs, and internal tools running under roughly 10 queries per second get more from throughput and price than from TTFT. Optimizing these for latency spends engineering time buying milliseconds nobody perceives.
The market prices this distinction explicitly: Fireworks sells batch inference at 50% of its serverless rates (fireworks.ai/pricing, fetched July 7, 2026), because latency-insensitive work batches aggressively and costs less to serve. The same logic applies inside a product: route summarization jobs and nightly re-embedding through the batched path, and reserve latency-optimized serving for the conversational loop where TTFT is the user experience.
What do these levers look like run as a product?
Inworld's Realtime Inference engine runs four of these levers as a product for consumer-scale realtime workloads: vLLM with a custom FlashInfer kernel patch (kernels), speculative decoding, KV cache optimization (caching), and NVFP4 quantization on NVIDIA B200 GPUs, serving open models including Gemma 4 and the DeepSeek V3.2/V4 family (
hosting guide).
The economics follow from the engineering: the same optimizations that cut TTFT raise tokens per GPU, which is why these first-party-hosted open models run at up to 50% below public third-party rates (
consumer AI cost announcement, June 10, 2026). The
Router routes to 220+ models at cost, with pay-as-you-go entry and spend-based discounts. We serve open models on our own engine; we do not train LLMs. The Router is OpenAI-SDK compatible (base URL
https://api.inworld.ai/v1), so benchmarking it against your current provider is a config change, not an integration project.
About Inworld AI
Inworld is a research lab and inference provider focused on realtime AI models for consumer-facing applications. We build first-party voice models (Realtime TTS and Realtime STT), serve optimized open-source LLMs on our own Realtime Inference engine, and expose them as modular APIs, alongside an LLM Router that routes to 220+ models and a Realtime API for full speech-to-text-to-LLM-to-speech pipelines. We focus on serving developers of realtime, high-volume conversational products across domains such as health, fitness, education, companions, social, and games, with an emphasis on quality, low latency, and low cost at scale.
Frequently asked questions
Does speculative decoding change model outputs?
No. A small draft model proposes several tokens ahead and the large target model verifies them in one parallel forward pass, accepting the longest matching run. The output distribution is provably identical to running the target model alone (
Leviathan et al., 2022). The gain is wall-clock speed, typically 2 to 3x on predictable text like code, structured output, and stable-persona dialogue, at the cost of extra GPU memory and compute for the draft model.
Does quantization reduce model quality?
It can, but well-calibrated low-precision formats keep the loss marginal. FP8, INT4, and NVIDIA's
NVFP4 4-bit floating-point format with two-level scaling shrink memory footprint and bandwidth pressure, so one GPU holds a bigger model or batch and serves more tokens per dollar. The honest trade-off: quality impact varies by model and task, so evaluate on your own workload before and after. Inworld's published stack runs NVFP4 on B200 GPUs for Gemma 4 serving.
Should I optimize P50 or P99 latency?
Measure both, and treat them as different problems. P50 (median) describes the typical request and responds to model choice, quantization, and caching. P99 describes your worst experiences and responds to queueing, batching-window tuning, and disaggregated prefill/decode, because tail latency usually comes from contention, not from the model. A consumer app under viral load lives at P99: that is the latency your loudest users actually see.
When is inference latency optimization premature?
When no human is waiting on the response. Batch enrichment jobs, overnight evaluation runs, and internal tools serving under roughly 10 queries per second get more value from throughput and price than from TTFT. Providers price this explicitly: Fireworks offers batch inference at 50% of serverless rates (fireworks.ai/pricing, July 7, 2026). Take the discount for offline work and spend latency effort on the user-facing path.
Next steps