Report: The Consumer AI Stack

Inworld Research | 2026

The AI industry invested $581 billion in 2025. It built the models, the data centers, the chips. What it did not build is economics that work for the applications people actually use.

IBM lost $70 billion in market value in a single session this month because customers are redirecting budgets to AI infrastructure, not AI products. Uber burned through its entire 2026 AI budget by April. A Flexera report found only 31% of IT leaders have accurate visibility into what they are spending on AI. Meta expanded a single data center project from $10 billion to $50 billion. The Federal Reserve cited AI infrastructure as one of three drivers of inflation.

This is the supply side. On the demand side, consumer AI reached a billion monthly users in three years, faster than the PC or the internet. But 97% of those users pay nothing. The entire GenAI mobile app market generated $3 billion in 2025. That is 0.5% of the infrastructure investment behind it.

Somewhere between $581 billion in and $3 billion out, the math has to work. It has to work inside the architecture of each session: the speech-to-text, the reasoning, the text-to-speech, the routing between them, the orchestration that holds the pipeline together. That is where cost is decided. Not at the model level. At the stack level.

One company found that out the hard way, and fixed it.

Wishroll's companion app, Status, hit 1 million users in 19 days. One of the fastest consumer app launches ever recorded. The product worked. People liked it. They came back. And the AI inference bill was on track to make the company insolvent within months.

The team did not switch to a worse model. They did not cut features. They restructured how the stack processed each conversation. One large, expensive prompt that handled everything (intent classification, safety checks, personality, response generation) was decomposed into small, task-specific models routed dynamically. Simple turns went to cheap models. Complex turns went to expensive ones. Total cost dropped 95%. Engagement held.

That is what this guide is about. Not the model. The architecture around the model. The choices about how to route, how to stream, how to handle interruptions, how to manage context across a two-hour session, and how to observe the relationship between AI decisions and user outcomes. Those choices determine your latency, your cost, and whether you can change the experience without rebuilding the pipeline.

The AI cost problem is not a pricing problem. It is an architecture problem. And it shows up first and hardest in consumer, where the margin between a product that scales and one that goes insolvent is measured in the milliseconds and microdollars of every turn.

A consumer voice AI app runs a loop. The user talks. The system listens, transcribes, thinks, and speaks back. Each pass through the loop is a turn. A companion session might run 200 turns. An enterprise support call runs 5-10.

Seven layers execute this loop. Each one adds latency, cost, and engineering surface area.

Client. The user's device captures audio, detects when they are speaking versus silent (Voice Activity Detection), and streams the audio to the server. VAD is the first latency decision. Silero, the current standard for client-side VAD, runs at about 30ms per audio frame. Total VAD latency: 50-100ms. The tension is between speed (detecting end-of-speech quickly) and accuracy (not cutting the user off mid-sentence). There is no setting that works well for every user in every environment.

Transport. Audio streaming requires persistent connections (WebRTC or WebSocket), not HTTP request-response. For a global user base, edge routing (placing inference as close to the user as possible) cuts network latency by 20-80ms. Edge presence in three regions (US, EU, APAC) keeps transport under 50ms for most users. The transport layer also handles what happens when mobile networks misbehave: dropped packets, WiFi-to-cellular handoffs, reconnection without losing session state.

Speech-to-text. Streaming transcription: the system begins producing partial transcripts while the user is still talking. Adds 100-300ms. In consumer applications, STT has to handle accented speech, background noise from bedrooms and kitchens and buses, overlapping speech during barge-in, and emotional signals (tone, cadence, volume) that should inform the AI's response. Inworld's STT ranks top-4 on independent benchmarks and includes realtime emotion detection, passing the user's emotional state downstream to the LLM and TTS.

Reasoning (LLM). The model that decides what the AI says. Critical metric: time-to-first-token (TTFT). Small models (7-13B parameters) deliver 50-100ms. Mid-size (30-70B) deliver 100-300ms. Frontier models (GPT-4o class) deliver 200-500ms. When your total round-trip budget is 800ms, a frontier model can consume more than half of it just reaching the first token. Context management is the other challenge: companion sessions run 50-200+ turns, and appending every turn to the prompt exceeds context windows and inflates costs. Production systems use context compression, selective memory retrieval, and hierarchical memory.

Text-to-speech. In a streaming pipeline, synthesis begins as soon as the LLM's first tokens arrive, delivering audio incrementally. The user hears the response start while the LLM is still generating the rest. This is what makes sub-800ms round-trip possible. Consumer TTS needs more than clarity: companion apps need emotional modulation. Learning apps need accent accuracy across languages. Inworld's Realtime TTS-2 (top-ranked on the Artificial Analysis Speech Arena) supports steering (voice direction in plain English, not a fixed emotion enum) and conversational context (the TTS hears the prior audio of the exchange and adapts delivery accordingly, not just the current sentence). One voice works across 100+ languages. The TTS leaderboard reshuffles monthly as new models enter. What does not reshuffle is the architecture underneath: consumer apps need the TTS to work inside a pipeline with STT, LLM routing, and orchestration, not as a standalone model. The quality of the voice matters. The cost and latency of the full session matters more.

Orchestration. The layer connecting everything into a working pipeline. Manages flow: when to start STT, when to forward partial transcripts, when to begin TTS, how to handle interrupts, how to route between models, how to run A/B tests on live traffic. Inworld's orchestration runs on a C++ core. Developers compose pipelines from production-ready building blocks rather than building from scratch. The orchestration layer is free; cost comes from model consumption.

Observability. In enterprise AI, observability means uptime and error rates. In consumer AI, it means something more specific: which model variant produced higher engagement? Which voice led to longer sessions? Which prompt wording improved retention? Inworld's platform connects telemetry to live experimentation, so infrastructure becomes a compounding advantage rather than a fixed cost.

Fully managed. A single platform handles the entire pipeline through one API. Inworld's Speech-to-Speech API is an example: handles turn-taking and interruption natively, integrates any major LLM, deploys a conversational endpoint in about three minutes.

Lowest engineering effort. Highest per-minute cost (managed orchestration on top of model consumption). But often the lowest total cost because you are not paying engineers to build and maintain the pipeline.

Use it from zero to 100K users, or for any app where the voice interaction follows standard conversational patterns.

Hybrid. Build your own orchestration. Use managed APIs for each model layer. You own the glue code. A team of 2-3 engineers can build this in 4-8 weeks. You can swap components independently, insert custom processing between stages, and optimize for your workload. The Inworld Router fits here: one API, 220+ models, no markup on third-party models (typical gateways add ~5%), switch models with a config change.

Use it at 100K-1M DAU, or when you have requirements the managed API does not cover.

Vertically integrated. Self-host models on your own GPUs. Own the full stack. Requires 5-10+ dedicated ML infrastructure engineers. Per-minute cost is lowest at scale: dedicated GPUs from $5/GPU-hour on Inworld compute versus $11/GPU-hour on GCP. At high utilization, 50-90% cheaper than API-based inference. Below roughly 1 million voice minutes per month, the operational overhead exceeds the savings.

Use it above 1M DAU, or when inference cost is a top-three business concern.

The crossover points. Managed to hybrid: when monthly API spend exceeds ~$50K and you have engineering capacity. Savings from smarter routing and reduced waste typically exceed engineering cost within 3-6 months. Hybrid to vertically integrated: when monthly inference spend exceeds ~$200K and you can sustain 60%+ GPU utilization. Below 60%, fixed GPU costs exceed variable API costs. Self-hosted STT specifically: Whisper is mature and runs on modest hardware. The crossover from managed STT to self-hosted happens around 500-1,000 hours of audio per month.

The total round-trip budget for a natural-feeling voice conversation is sub-800ms from end-of-user-speech to start-of-AI-audio.

Sequential: 280-1,350ms. The worst case is nearly 5x the best case.

Streaming (each component processes the previous one's output as it arrives): 200-600ms.

This is why a model choice that adds 200ms of LLM latency versus one that adds 100ms is not a small tradeoff. That 100ms is paid on every turn. Over 200 turns, it is the difference between a conversation that flows and one where the user consciously waits for every response.

Barge-in is when the user talks over the AI. In enterprise voice (support calls, IVR), it happens occasionally. In companion conversations, it happens on 15-30% of turns. Users agree, redirect, react, add context mid-thought. That is how conversations work.

Handling it well is the hardest real-time engineering problem in consumer voice AI, and the primary reason off-the-shelf solutions feel robotic.

Three approaches.

Cancel-and-restart. Detect user speech, stop everything, process new input from scratch. Simple to build. Wasteful: every token from the interrupted response is thrown away. Contextually messy (was the partial response part of the conversation?).

Graceful interrupt. Detect user speech, fade out audio over 100-200ms, note how much the user heard, process new input, generate a response that accounts for the interruption. More complex. Much more natural. This is the production standard.

Predictive silence detection. Analyze the user's audio for signals they are about to speak (breath intake, pause patterns) and preemptively slow the AI or insert natural pauses. Few systems do this in production. It requires parallel low-latency audio analysis.

The hidden cost of all three: in a streaming pipeline, the TTS synthesizes audio from the LLM's tokens as they arrive. When the user interrupts, tokens generated after the last chunk the user heard are wasted. In typical companion conversations, 15-30% of generated TTS audio is never heard. That waste is invisible in benchmarks and real on production bills.

Three ways to reduce it. Aim for conversational-length responses (not paragraphs). Detect user intent to speak earlier (cancel generation before too many tokens are wasted). Use a cheap model for initial tokens, escalating to a higher-quality model only if the user is listening. The Inworld Router enables that kind of dynamic routing.

Running one frontier model for every conversational turn is the most expensive architecture pattern in consumer AI. A GPT-4o processing "I see, tell me more" costs the same per token as a GPT-4o processing a complex reasoning question. The simple response does not need a frontier model. It needs a 7B parameter model that costs a fraction and responds in a fraction of the time.

A typical companion turn involves several tasks: intent classification (what is the user doing?), safety filtering (does this violate policies?), response generation (the actual reply), personality injection (matching the companion's character), and memory retrieval (pulling relevant history). In a monolithic architecture, all of this happens in one prompt sent to one large model. In a decomposed architecture, each task goes to the smallest model capable of handling it. Only the tasks that genuinely need a large model get one.

Wishroll used exactly this approach. 95% cost reduction. Engagement held. The Inworld Router provides the mechanism: 220+ models from every major lab through one API, no markup on third-party models, model switching via config change, A/B testing on live traffic.

When routing pays for itself: below 100K daily users, the complexity may not justify the savings. A single mid-tier model handles everything at reasonable cost. At 100K-500K, the volume of simple turns (which a small model handles at 5-10x lower cost) is large enough that aggregate savings exceed the engineering investment. Above 500K, routing is not optional. It is the difference between a sustainable AI bill and one that makes the business unviable.

0 to 10K DAU. Use managed. One API, one model, one voice. Focus on whether people want the product. Cost per user is high but total spend is $1K-$10K/month. Do not build infrastructure. Ship.

10K to 100K DAU. Cost enters the top five business concerns. Track cost per session. Basic routing (separate intent classification from response generation). Volume-tier pricing. Basic observability. Spend: $10K-$100K/month.

100K to 500K DAU. Move to hybrid. Custom orchestration. Sophisticated routing via the Inworld Router. Persistent memory. Multi-region deployment. Proper barge-in handling. Live A/B testing. Spend: $100K-$500K/month. The routing/non-routing difference at this scale can be $200K-$400K per month.

500K to 1M+ DAU. Evaluate dedicated compute. Fixed GPUs beat variable API costs above 60% utilization. Three or more regions. Predictive scaling. Advanced context management. Continuous experimentation. Spend: $500K-$2M+/month.

The question is not "should we build our own TTS?" Almost certainly not. The question is how much of the stack around the models you should own.

If your competitive advantage is the AI experience (conversation quality, emotional expressiveness, response speed), own more of the stack. More levers. If your advantage is product design, content, or community, use a managed stack and focus engineering where you differentiate.

Most consumer AI apps should start managed, move to hybrid around 100K DAU when cost becomes material, and add vertically integrated elements only for specific components where the savings or differentiation at their volume justify the engineering. The Inworld platform supports that progression. Speech-to-Speech API at the start. Router for model optimization as you scale. Realtime Inference and dedicated Compute when you need maximum cost efficiency. The stack grows with the app.

The AI industry is spending at a pace that dwarfs anything in the history of technology. $581 billion in one year. Data centers at 5 gigawatts. The supply side is not the constraint. The constraint is on the other side: applications that create enough value for people to pay for them, at a cost that does not consume the revenue they generate.

That constraint lives in the stack. Not in the model. In the seven layers that execute every turn of every conversation, and in the choices about how those layers connect: which model handles which task, how context flows between them, how latency is managed across regions, how the pipeline handles a user who interrupts, and how the economics work when the session runs for an hour and the user pays five dollars a month.

The companies that solve this will not be the ones with the cheapest model on a benchmark. They will be the ones with an architecture where the cost of each session falls as the application scales, rather than rising in lockstep with engagement. Where the infrastructure becomes a compounding advantage rather than a fixed tax.

That is what this guide maps. The blueprint is here. The rest is execution.

Macro: Stanford HAI AI Index 2026 ($581B global AI investment), IBM Q2 2026 earnings (July 2026), Forbes/Flexera (31% AI spend visibility, July 2026), Meta Hyperion expansion (July 2026), Federal Reserve Governor Waller remarks (July 2026). Consumer market: Sensor Tower State of AI 2026, Business of Apps, Menlo Ventures 2025 State of Consumer AI (3% pay rate), DataReportal Digital 2026. Architecture and scaling: Inworld engineering practices and anonymized customer patterns. Latency: Coval, Gradium, MorVoice, Picovoice. Cost: provider pricing (June-July 2026), Inworld June 2026 update. Customers: Wishroll (~95% cost reduction, public), Talkpal, Bible Chat, Luvu (all public Inworld customers; figures self-reported). Barge-in: Inworld production telemetry (aggregated, anonymized). All data Q2 2026.