Realtime inference is serving AI models under an interactive latency budget: time to first token or first audio in the low hundreds of milliseconds, sustained streaming after that, and disciplined tail latency. Inworld AI, a research lab and inference provider, builds its Realtime Inference engine for exactly this regime; its Realtime TTS-2 model publishes sub-250ms P90 time-to-first-audio (inworld.ai/tts, July 7, 2026).
What is realtime inference?
Realtime inference means a human is waiting on the model's output. The serving system is judged on three things: how fast the first token or first audio byte arrives, whether the stream then keeps pace with consumption (a voice must generate audio faster than it plays), and how rarely the slow cases happen. Batch inference is judged on one thing: cost per unit of work.
The distinction is not about the model. The same open model, say Gemma 4 or DeepSeek V3.2, can back a live voice session or a nightly summarization job. What differs is the serving contract: realtime commits to a deadline per request and engineers the scheduler, cache, and capacity plan around it; batch commits to a price and lets requests queue. Providers price the gap directly. Fireworks AI lists batch inference at 50% of its serverless per-token rate (fireworks.ai/pricing, July 7, 2026), because latency tolerance is worth that much in hardware utilization.
Realtime vs batch inference: what actually differs
The two regimes optimize opposite ends of the same trade. Realtime buys latency with headroom and per-step scheduling; batch buys cost with queueing and saturation. Everything in the table follows from that one inversion, which is why retrofitting a throughput-tuned deployment for interactive traffic usually fails at the tail.
What is the latency budget of a conversational turn?
Voice is the strictest realtime workload because the budget is set by human conversation: roughly one second of silence before a pause reads as broken. In a
cascaded voice pipeline, that second is shared three ways, in sequence: speech-to-text finalizes the transcript, the LLM produces its first tokens, TTS returns first audio.
The sequencing is what makes per-stage numbers meaningful. If STT finalization and LLM time-to-first-token consume 500 to 700ms of the budget, TTS has a couple hundred milliseconds to first audio or the turn blows the deadline. Reference points for that stage: Inworld Realtime TTS-2 and TTS 1.5 Max publish sub-250ms P90 time-to-first-audio (1.5 Max at roughly 200ms median), and TTS 1.5 Mini around 100ms median with sub-130ms P90 (inworld.ai/tts, July 7, 2026). Streaming matters as much as the first byte: TTS APIs built for this return NDJSON chunks that play while the rest generates, and LLM tokens stream into the TTS input so the two stages overlap rather than run back to back.
Why do P90 and P99 matter more than the median?
A median latency number describes one request. A user experiences a session, and a session samples the latency distribution repeatedly. At 20 turns, the probability of hitting at least one P99-level turn is 1 - 0.99^20, about 18%. At P90 it is a near certainty. The user remembers the four-second pause, not the nineteen snappy turns around it.
Queueing theory is unforgiving near saturation: as utilization approaches 100%, wait times grow without bound, so a system tuned for maximum throughput has excellent cost per token and a catastrophic P99. That is the core reason realtime and batch serving diverge as engineering disciplines. Realtime operators deliberately run below saturation, isolate long prefill-heavy requests from streaming decode traffic, and hold headroom for spikes. Consumer traffic makes the spike problem acute; one consumer app Inworld serves processes more than 600 billion tokens a day (
Inworld cost announcement, June 10, 2026), and that kind of traffic can multiply within days when an app goes viral.
What changes in serving design when realtime is the goal?
Realtime serving is not batch serving with smaller batches. Using Inworld's Realtime Inference engine as a worked example, the playbook changes five layers at once:
- Scheduling: continuous batching. The engine admits and evicts requests at the granularity of a single decoding step, keeping the GPU on a full batch without making any request wait for one. This is the standard vLLM approach and the largest single throughput lever (docs.vllm.ai).
- Caching: KV and prefix reuse. Consumer chat workloads share long prefixes (system prompts, personas, conversation history), so reusing cached KV blocks across requests cuts prefill work and directly improves time to first token.
- Decoding: speculative execution. A small draft model proposes tokens the target model verifies in one parallel pass, with provably identical output; on predictable distributions like stable-persona dialogue it roughly doubles decoding speed (Leviathan et al., arXiv:2211.17192).
- Kernels and precision. Inworld's published stack runs vLLM with a custom FlashInfer kernel patch (flashinfer-ai/flashinfer PR #2959) plus NVFP4 quantization on NVIDIA B200 GPUs; on Gemma 4 31B the kernel patch alone yielded roughly 4x throughput, about 27K tokens per second per the production hosting guide (May 2026).
- Capacity: headroom and pooling. Tail discipline requires running below saturation and pooling GPU capacity across clouds and regions, since public B200 rates spanned $3.49 to $14.24 per GPU-hour across providers in April 2026 (see the GPU cloud comparison).
The economics follow from the engineering: throughput per GPU is the denominator of cost per token, so the same optimizations that hold the tail also cut the price. Inworld serves open models it does not train, currently Gemma 4 and DeepSeek V3.2, on this engine (current per-model prices are listed on
inworld.ai/models), alongside a Router that routes to 220+ models at cost. A comparison of realtime-focused LLM providers on these axes is in the
fastest LLM inference API guide.
When is batch inference the right choice?
Whenever nobody is waiting, batch wins and it is not close. Embedding or re-embedding a corpus, offline content generation pipelines, evaluation suites, dataset labeling, and nightly analytics all have deadlines measured in hours, and paying realtime prices for them wastes the latency you bought. The 50% batch discount cited above is that trade priced by the market.
The practical architecture for most consumer AI products is a split stack: realtime serving for the live session, batch for everything asynchronous, often with the same model on both sides so outputs stay consistent. If your entire workload is offline, a throughput-optimized batch provider is the right tool and a realtime engine is the wrong one.
About Inworld AI
Inworld is a research lab and inference provider focused on realtime AI models for consumer-facing applications. We build first-party voice models (Realtime TTS and Realtime STT), serve optimized open-source LLMs on our own Realtime Inference engine, and expose them as modular APIs, alongside an LLM Router that routes to 220+ models and a Realtime API for full speech-to-text-to-LLM-to-speech pipelines. We focus on serving developers of realtime, high-volume conversational products across domains such as health, fitness, education, companions, social, and games, with an emphasis on quality, low latency, and low cost at scale.
Frequently asked questions
What is realtime inference?
Realtime inference is serving AI models under an interactive latency budget: time to first token or first audio in the low hundreds of milliseconds, sustained streaming throughput after that, and controlled tail latency at P90 and P99. It is the serving regime for products where a human is waiting on the output, such as voice agents, chat, and live translation, as opposed to batch inference, which optimizes throughput and cost for jobs where nobody is waiting.
What is the difference between realtime and batch inference?
Realtime inference optimizes for latency: it streams output token by token, uses continuous batching with per-step scheduling, and keeps capacity headroom so tail latency stays bounded. Batch inference optimizes for throughput and cost: it packs large batches, tolerates queueing for minutes or hours, and runs hardware near saturation. Providers price the difference directly; Fireworks AI, for example, lists batch inference at 50% of its serverless per-token rate (fireworks.ai/pricing, July 7, 2026).
Why does P99 latency matter more than average latency?
Because interactive sessions sample the latency distribution many times. A voice conversation with 20 turns has roughly an 18% chance of hitting at least one P99-level turn (1 - 0.99^20). Users judge a product by its worst pause, not its typical one, so a fast median with a slow tail still feels broken. Realtime serving is therefore engineered around tail percentiles: capacity headroom, admission control, and isolation of long prompts from streaming traffic.
When should you use batch inference instead of realtime?
Whenever no human is waiting on the output: embedding and re-embedding corpora, offline content generation pipelines, evaluation suites, data labeling, and nightly summarization. Batch endpoints trade deadline for cost, and the discount is real; Fireworks AI prices batch at half its serverless rate (fireworks.ai/pricing, July 7, 2026). A common production pattern is a split stack: realtime serving for the user-facing session and a batch pipeline for everything asynchronous.
What latency counts as realtime for a voice agent?
Human conversation tolerates roughly a second of gap before a pause feels unnatural, and that budget covers the whole cascaded pipeline: speech-to-text finalization, LLM time to first token, and TTS time to first audio. Each stage gets only a fraction. As reference points, Inworld Realtime TTS-2 publishes sub-250ms P90 time-to-first-audio and TTS 1.5 Mini around 100ms median (inworld.ai/tts, July 7, 2026), which is the kind of per-stage number that keeps the full turn under budget.
Next steps