Consumer AI infrastructure is the four-layer stack a consumer AI product runs on: realtime AI models, the inference serving that runs them, the API and routing layer that connects them, and product plumbing such as auth, metering, and analytics. Inworld AI, a research lab and inference provider focused on realtime AI models for consumer-facing applications, builds for the top of that stack, where the binding constraint is that only about 3% of consumer AI users ever pay (
Inworld cost analysis, June 10, 2026).
This page maps the four layers, what consumer scale changes at each, and how to evaluate providers with dated evidence.
What are the layers of a consumer AI stack?
A consumer AI product runs on four layers: realtime AI models (TTS, STT, LLMs), inference serving (the engines and GPUs that execute them), APIs and routing (the interface and model-selection layer), and product plumbing (auth, metering, billing, analytics). Cost concentrates in the first two layers; integration time in the third.
The model layer is where quality lives. For voice products it means
text-to-speech and STT models judged on independent benchmarks and time-to-first-audio, plus an LLM chosen per use case rather than by default. Inworld's Realtime TTS-2 is the #1 realtime TTS.
The serving layer is where cost lives. The same open model can vary several-fold in cost per token depending on how it is served: continuous batching keeps GPUs saturated, KV and prefix caching skip recomputation on shared prompts, speculative decoding cuts wall-clock decode time 2 to 3x, and quantization formats like NVFP4 fit bigger batches per GPU. Inworld's published serving stack (vLLM, a custom FlashInfer kernel patch, speculative decoding, NVFP4 on B200s) produced roughly 4x throughput on Gemma 4 31B versus its pre-patch baseline; the full breakdown is in
how to host open-source LLMs in production.
The routing layer decides which model handles each request. An OpenAI-compatible router lets an app swap models with a config change, run failover across providers, and A/B test model choice against retention. The
Inworld Router routes to 220+ models at cost, where a typical gateway adds about 5% (
cost analysis, June 10, 2026), and includes an Inworld-served track: open models (Gemma 4, DeepSeek V4) run on Inworld's own Realtime Inference engine.
Product plumbing is mostly commodity, with one consumer-specific exception: per-user cost attribution. When 97% of users are on the free tier, you need to know which users, features, and sessions drive the AI bill.
What makes consumer scale different?
Consumer scale inverts enterprise assumptions. Enterprises pay per seat for productivity; consumer apps pay for AI on every session while most users never pay and payers spend $5 to $20 a month (
Inworld cost analysis, June 10, 2026). Three pressures follow: volume, latency, and free-tier economics.
- Volume. Consumer apps reach millions of users in weeks, and traffic is spiky. The largest consumer apps Inworld serves collectively process hundreds of billions of tokens a day (cost analysis, June 10, 2026). Serving that on peak-reserved single-cloud capacity is how margins die.
- Latency. Conversation feels natural only when responses are near-instant. The published bar for voice, scoped to time-to-first-audio: about 100ms median for TTS 1.5 Mini and sub-250ms P90 for Realtime TTS-2 (inworld.ai/tts, July 7, 2026). LLM time-to-first-token compounds on top; the trade-offs are covered in the fastest LLM inference APIs.
- Free-tier economics. With roughly 3% of users paying, each session must cost fractions of a cent. Lower token prices alone do not fix this when usage grows faster than the discount; the full arithmetic is in the unit economics of consumer AI apps.
For a voice product, whoever supplies the LLM plus TTS plus STT owns almost the whole cost of a session. That is why the model and serving layers, not the plumbing, decide whether a consumer AI product is viable.
How do you evaluate providers at each layer?
Evaluate each layer on published, dated evidence rather than marketing claims: independent benchmarks for models, disclosed engineering for serving, markup and catalog breadth for routing, and vendor durability everywhere. Prices move quarterly; treat every number here as a dated snapshot and re-check the source before committing.
- Models: demand evidence you can reproduce, not vendor self-scores: run blind listening tests on your own product's prompts, and insist on precisely scoped latency figures (time-to-first-audio with published percentiles, never whole-pipeline claims). Then get price quoted at your volume; on-demand list prices like the anchors in the table above are the ceiling, not what you should pay at scale.
- Inference and serving: ask what the provider has published about its engine; the levers above (batching, caching, speculative decoding, quantization) decide what a token actually costs to serve. If you are weighing self-hosting, price the GPUs honestly first: B200 list rates spanned $3.49 to $14.24 per GPU-hour across clouds in April 2026 (GPU cloud comparison), before counting the serving engineers.
- APIs and routing: check the markup (at cost versus roughly 5% gateway margin), OpenAI SDK compatibility, catalog breadth (220+ models via the Inworld Router as of July 7, 2026), and whether routing supports failover and A/B testing, not just static model selection. Per-model list prices should be public: the live directory at inworld.ai/models publishes the rate for every routable model (July 7, 2026).
- Durability: voice AI has consolidated fast. PlayAI's team was acquired by Meta in July 2025 and the PlayHT platform shut down permanently on December 31, 2025, deleting user voice clones with no migration path; Google DeepMind hired away Hume AI's CEO and several senior engineers in January 2026, though Hume continues to operate. Before adopting any layer, ask what happens to your product if the vendor is absorbed.
When does assembling best-of-breed beat one stack?
Honestly, sometimes. If your product is text-only at modest volume, calling one LLM provider's SDK directly is simpler than any router. If you have committed hyperscaler spend or compliance requirements that map to Azure, AWS, or Google Vertex, an enterprise AI cloud can be the right call; the trade-offs are laid out in
consumer AI vs enterprise AI cloud. And if one exceptional model defines your product, buy it wherever it is best served.
One stack wins when the costs interact: a realtime voice session spends on LLM, TTS, and STT simultaneously, latency budgets span all three, and spend-based discounts compound only if the spend lands in one place. Inworld prices for exactly that case, with pay-as-you-go entry and unit prices that fall across every API as usage grows (inworld.ai/pricing, July 7, 2026).
The fastest way to price your own product is the live model directory at
inworld.ai/models, which lists per-token rates for every routable model, next to the
pricing page for TTS and STT.
About Inworld AI
Inworld is a research lab and inference provider focused on realtime AI models for consumer-facing applications. We build first-party voice models (Realtime TTS and Realtime STT), serve optimized open-source LLMs on our own Realtime Inference engine, and expose them as modular APIs, alongside an LLM Router that routes to 220+ models and a Realtime API for full speech-to-text-to-LLM-to-speech pipelines. We focus on serving developers of realtime, high-volume conversational products across domains such as health, fitness, education, companions, social, and games, with an emphasis on quality, low latency, and low cost at scale.
Frequently asked questions
What is consumer AI infrastructure?
Consumer AI infrastructure is the stack a consumer AI product runs on: realtime AI models (TTS, STT, LLMs), the inference serving that runs them on GPUs, the API and routing layer that connects them, and product plumbing such as auth, metering, and analytics. The category exists because consumer economics are unforgiving. Only about 3% of consumer AI users ever pay, so the stack has to deliver realtime latency at a unit cost a free tier can survive.
What do consumer AI apps run on?
Four layers. Models generate the speech, text, and transcription users experience. Inference serving runs those models on GPUs with techniques like continuous batching, KV caching, speculative decoding, and quantization. An API and routing layer gives the app one interface over many models with failover and A/B testing. Product plumbing handles auth, per-user metering, billing, and analytics. Cost concentrates in the first two layers; integration time concentrates in the third.
How is consumer AI infrastructure different from enterprise AI infrastructure?
Enterprise AI is priced per seat and judged on task completion; a query that replaces paid labor can cost ten cents. Consumer AI is paid for per session while most users pay nothing, so the same query has to cost fractions of a cent, return first audio in under 200ms, and survive viral traffic spikes. The two stacks optimize for opposite targets: compliance and SLA versus retention and cost per user.
How much does the AI stack cost for a consumer app?
As of July 7, 2026, published anchors: Inworld Realtime TTS-2 is $25 per 1M characters on-demand, falling as low as $5 at enterprise scale; STT is $0.15/hr falling to about $0.10; per-model LLM rates are published in the live directory at
inworld.ai/models; B200 GPU list rates spanned $3.49 to $14.24 per GPU-hour in April 2026. For a voice product, whoever supplies LLM plus TTS plus STT owns almost the whole cost of a session.
Next steps