LLM inference is expensive because every generated token consumes GPU time, and Inworld AI, a research lab and inference provider for realtime consumer applications, treats one variable as decisive: how many tokens each GPU-hour actually produces. Public NVIDIA B200 list rates spanned $3.49 to $14.24 per GPU-hour across clouds in April 2026.
This page covers the cost anatomy, the levers that move it, why volume improves unit economics, and what that means when you buy inference.
Why is LLM inference expensive?
Inference cost is GPU-time per token. A GPU-hour costs the same whether it emits 2,000 or 27,000 tokens per second, so cost per token is the hourly rate divided by realized throughput. Utilization is the whole game: an idle or half-loaded GPU produces expensive tokens, a saturated one produces low-cost tokens from identical hardware.
The cost anatomy breaks into three parts:
- Prefill (processing the prompt) is compute-bound and bursty. Long prompts monopolize compute in spikes and stall other users' token streams when both phases share GPUs.
- Decode (generating tokens one step at a time) is memory-bandwidth-bound and steady. A single request cannot come close to saturating a modern GPU, so serving requests one at a time wastes most of the silicon you are paying for.
- Idle capacity costs the same as busy capacity. Traffic peaks force you to provision for the spike, and every off-peak hour degrades your effective cost per token.
That is why two providers running the same open model on the same GPU can quote very different per-token prices. The hardware cost is public. The difference is engineering and load.
The gap is measurable. In a small benchmark run on 2026-07-07, Gemma 4 26B A4B served on Inworld's stack returned a median TTFT of 241 ms at 113 tokens per second, while the same model on a general-purpose host returned 445 ms at 17 tokens per second. Same weights, same request, different serving layer; the full table and methodology are on the
inference latency optimization page.
Which engineering levers actually cut inference cost?
Six levers account for most of the gap between naive and optimized serving. Each raises tokens per GPU-second or lowers the GPU price underneath, and they compound.
A seventh lever, disaggregated prefill/decode, runs the two phases on separate GPU pools and ships the KV cache between them, so compute-bound prefill stops stalling bandwidth-bound decode (DistServe, arXiv 2401.09670). It matters most at high load, exactly where unit cost is decided.
For the Inworld-specific application of this stack, see
how Inworld hosts open-source LLMs in production. For how these levers show up as latency rather than cost, see the
fastest LLM inference API comparison.
Why does volume improve unit economics?
Scale pushes cost per token down, not up. Aggregating many tenants smooths the load curve, and smoother load means higher average utilization from the same fleet. Volume also unlocks committed capacity at rates far below on-demand.
Three mechanisms drive it:
- Utilization smoothing. One consumer app has a brutal daily peak. Hundreds of apps across time zones and use cases fill each other's troughs, so the fleet runs closer to saturation around the clock.
- Committed capacity. Predictable aggregate volume justifies reserved and committed GPU pricing. Inworld's published dedicated-compute floor is $5 per GPU-hour, less than half the on-demand rate of a hyperscaler (Inworld cost blog, June 10, 2026).
- Cache economics. More traffic on the same models raises prefix-cache hit rates, and cached prefill costs dramatically less than recomputed prefill.
This is the engineering half of the argument made in
the consumer AI cost post (June 10, 2026): today's prices are the ceiling, not the floor, because scale makes the serving layer more efficient, not less. Per the same post, one consumer app Inworld serves processes more than 600 billion tokens a day (a customer-reported figure). At that volume, a few percent of utilization is real money.
What does this mean when you are buying inference?
A per-token list price is a proxy for two things you cannot see directly: the provider's fleet utilization and the quality of its serving engineering. A low price sustained at high reliability means the provider solved the problems above. A low price with throttling and timeouts means you are absorbing the gap.
Two buying models exist, and the distinction matters more than any single price:
- At-cost routing. Inworld's Router passes third-party models through at cost, no markup, where a typical gateway adds about 5% (cost blog, June 10, 2026). The routed model's underlying economics remain the upstream provider's.
- First-party serving. Where the provider owns the serving layer, the levers above land in the price. Inworld's Realtime Inference serves optimized open models up to 50% below the public third-party rate, per the same June 10, 2026 post.
The first-party track currently serves Gemma 4 26B A4B IT, Gemma 4 31B IT, and DeepSeek V3.2; current per-model prices are listed in the public model directory at
inworld.ai/models. Both tracks sit behind one OpenAI-compatible endpoint; the Router routes to 220+ models, with discounts across APIs that increase with total spend (
inworld.ai/pricing, as of 2026-07-07).
When does self-hosting beat any provider?
Sometimes it does. Self-hosting wins when utilization is steady and high enough that committed GPUs stay saturated around the clock, or when data constraints require model weights and traffic to stay inside your own perimeter. In those cases no provider margin, however thin, beats owned capacity.
The checklist:
- Steady, predictable, high-volume load. If committed GPUs would stay near saturation through the night and the trough, owned serving can undercut any per-token price.
- Hard data-residency or compliance constraints. Regulated workloads that cannot leave your VPC remove the provider option regardless of economics.
- Serving engineering as a competitive priority. The 3 to 4x throughput gap between stock and tuned deployments (production hosting guide, May 2026) is real; closing it yourself means staffing for kernels, batching, and capacity management permanently.
If your traffic is spiky or consumer-shaped (most sessions at peak hours, most users free), the utilization math usually favors an aggregated fleet. The
production hosting guide walks all three paths honestly, and the
GPU cloud comparison has the dated per-GPU rates for running the self-hosting math yourself.
About Inworld AI
Inworld is a research lab and inference provider focused on realtime AI models for consumer-facing applications. We build first-party voice models (Realtime TTS and Realtime STT), serve optimized open-source LLMs on our own Realtime Inference engine, and expose them as modular APIs, alongside an LLM Router that routes to 220+ models and a Realtime API for full speech-to-text-to-LLM-to-speech pipelines. We focus on serving developers of realtime, high-volume conversational products across domains such as health, fitness, education, companions, social, and games, with an emphasis on quality, low latency, and low cost at scale.
Frequently asked questions
Why is LLM inference so expensive?
Because every generated token consumes GPU time, and GPUs are costly to run: public NVIDIA B200 list rates spanned $3.49 to $14.24 per GPU-hour across clouds in April 2026. Decode is memory-bandwidth-bound, so a naively served model leaves most of the GPU idle while you pay for all of it. Cost per token is essentially GPU-hour price divided by tokens actually produced in that hour, which makes utilization the dominant variable.
What is the single biggest lever for reducing LLM inference cost?
Continuous batching. Admitting and evicting requests at the granularity of a single decoding step keeps the GPU working on a full batch as requests arrive and complete at different times. vLLM's continuous batching plus PagedAttention delivers order-of-magnitude throughput gains over naive serving, and every extra token per GPU-second is a direct reduction in cost per token. The other levers, caching, quantization, speculative decoding, and kernel work, compound on top of it.
Does quantization hurt model quality?
Done carelessly, yes. Calibrated low-precision formats keep accuracy loss minimal while cutting memory footprint and bandwidth pressure roughly in half versus FP8. NVIDIA's NVFP4 format uses 4-bit floating point with two-level scaling specifically to preserve accuracy on Blackwell GPUs. Inworld serves Gemma 4 with NVFP4 on B200s in production, per its published serving-stack writeup, and evaluates quality before and after quantization rather than assuming the format is free.
When does self-hosting cost less than an inference provider?
When your load is steady and high enough to keep committed GPUs at high utilization around the clock, or when data constraints require weights and traffic to stay inside your perimeter. Consumer traffic is rarely steady, and a specialized provider typically extracts 3 to 4x more throughput per GPU than a stock self-hosted deployment, so most spiky workloads come out ahead on a provider. Run the math on your own utilization curve before deciding.
How do you get started?