Cascaded vs Speech-to-Speech Voice Architecture

Last updated: July 7, 2026

Every voice agent uses one of two architectures: a cascaded pipeline that chains speech-to-text, an LLM, and text-to-speech, or an end-to-end speech-to-speech model that maps audio directly to audio. Neither is categorically faster: a well-engineered cascaded pipeline, where TTS streams first audio in roughly 100 to 250 ms (Realtime TTS 1.5 Mini ~100ms median, Realtime TTS-2 sub-250ms P90; inworld.ai/tts, July 7, 2026), can beat some end-to-end models on voice-to-voice latency. Inworld AI is a research lab and inference provider focused on realtime AI models for consumer-facing applications; the Inworld Realtime API is a configurable cascaded pipeline over a single stateful connection.

Below: how each architecture works, a component-level latency budget with dated sources, and a decision checklist. The goal is that you pick correctly for your product, not that you pick cascaded.

What is the difference between cascaded and speech-to-speech architecture?

A cascaded architecture runs three specialized models in sequence, with text at every boundary: STT transcribes the user, an LLM writes the reply, TTS speaks it. An end-to-end speech-to-speech architecture runs one multimodal model that consumes audio tokens and emits audio tokens, with no intermediate transcript required.

In a cascaded pipeline, the flow per conversational turn is:

Streaming STT emits partial transcripts while the user is still speaking.
A turn-detection layer (voice activity detection, ideally semantic VAD) decides the user has finished.
The final transcript goes to an LLM, which streams a text response token by token.
Streaming TTS converts the text stream to audio, starting playback before the LLM has finished writing.

Because every boundary is text, you can intercept it: run moderation, inject retrieved context, call tools, log the exact words both sides said. The Inworld Realtime API packages this pipeline behind one stateful WebSocket connection, so you configure the STT, LLM, and TTS components instead of stitching three vendors together. "Realtime" in the name refers to system responsiveness, not to an end-to-end speech model; the TTS API is a separate, standalone product you can use inside any stack. If you want to assemble the stages yourself, see build a cascaded STT to LLM to TTS pipeline step by step.

In an end-to-end model (the OpenAI gpt-realtime family and Gemini's native-audio Live API are the reference implementations), one network hears the audio and produces the response audio directly. The model perceives what a transcript destroys: tone, hesitation, sarcasm, laughter, accent, background context. Transcripts still exist, but as secondary outputs generated for logging, not as the substrate the model reasons over.

How do the two architectures compare?

Cascaded pipelines win on model choice, observability, cost control, and independent upgrades. End-to-end models win on paralinguistic nuance and stack simplicity. The right column of the matrix below lists real advantages, not strawmen.

Cost illustration, all figures dated July 7, 2026: on the cascaded side, Inworld STT is $0.15/hr on demand ($0.0025/min) and TTS-2 is $25 per 1M characters on demand (inworld.ai/pricing). At a typical 700 to 900 characters per minute of English speech (camb.ai TTS price comparison, fetched 2026-07-07), agent speech costs roughly $0.02 per spoken minute on TTS-2, before the LLM. LLM cost is whatever your chosen model costs, routed at cost with no markup across 220+ models via the Inworld Router. On the end-to-end side, OpenAI lists gpt-realtime-2.1 at $32 per 1M audio input tokens and $64 per 1M audio output tokens, and gpt-realtime-2.1-mini at $10/$20 (developers.openai.com pricing, fetched 2026-07-07); Google dual-quotes Gemini 3.1 Flash Live (a preview model) at $0.005/min audio in and $0.018/min audio out (ai.google.dev pricing, fetched 2026-07-07). The structural difference matters more than any single number: in a cascaded stack you can downgrade the LLM, cache prompts, or shorten responses to cut cost per component; in an end-to-end model the price is a package.

Is a cascaded pipeline slower than an end-to-end model?

No. This is the most persistent myth in voice agent architecture. Latency is a sum of component budgets, and the historical bottleneck, TTS synthesis, has collapsed: Inworld's Realtime TTS 1.5 Mini streams first audio at roughly 100ms median, and Realtime TTS-2 at sub-250ms P90 (inworld.ai/tts, July 7, 2026). A well-engineered cascaded pipeline achieves lower latency than some end-to-end speech-to-speech models.

Here is the component-level budget for a cascaded turn. Figures marked "vendor claim" are the provider's published numbers, not independent measurements; figures marked "planning budget" are engineering estimates you should validate against your own stack.

Two things follow from this table. First, the LLM dominates. If your voice-to-voice latency is bad, the fix is almost never "abandon the cascade"; it is a faster LLM, a shorter prompt, or better endpointing (see how semantic VAD handles turn-taking in both architectures). Second, the cascade's stages overlap when everything streams: STT transcribes while the user talks, TTS starts speaking on the LLM's first sentence. A cascade only feels slow when it is built non-streaming, stage by stage.

End-to-end models avoid two text-boundary conversions, which is a real structural advantage. But they run one large multimodal forward pass per turn, and vendors publish audio-token prices, not voice-to-voice latency medians, so there is no dated public figure to put in the right-hand column. Measure both on your own traffic before believing either side's marketing, including ours.

To test these numbers on your own stack, get an API key at platform.inworld.ai and measure time to first audio yourself with the TTS quickstart; the free tier includes up to 70 minutes of TTS.

When is end-to-end speech-to-speech the better choice?

Choose an end-to-end model when expressivity transfer is the product. If your agent must react to how something was said (a companion that hears the user's flat tone and responds gently, a language tutor that catches mispronunciation, a social app that laughs when the user laughs), the transcript boundary in a cascade destroys exactly the signal you need. No pipeline reconstruction fully recovers it.

End-to-end is also the pragmatic choice when:

You want single-vendor simplicity. One connection, one SDK, one bill, one throat to choke. For a prototype or a small team, gpt-realtime-2.1 or Gemini native audio is genuinely less engineering than any pipeline, managed or not.
Your turns are short and conversational. The paralinguistic advantage is largest in chit-chat and emotional registers, and the audio-token cost premium matters least when responses are brief.
You are already committed to that vendor's LLM. If the reasoning model you would pick anyway is the same vendor's frontier model, the cascade's model-choice advantage shrinks.
Voice cloning and voice identity are not core requirements. End-to-end models ship with fixed voice sets; if any pleasant voice works, this constraint costs you nothing.

For a survey of the end-to-end options, see the best speech-to-speech APIs compared.

When is a cascaded pipeline the better choice?

Choose a cascade when you need control: over which models run, what they cost, what gets logged, and what happens between hearing and speaking. For production voice agents in regulated or high-volume consumer domains such as health, education, support, and companion apps, these constraints usually decide the architecture before latency is even discussed.

Concretely, the cascade wins when:

You need the best model per component. The strongest reasoning LLM, the fastest STT for your accent mix, and the voice your brand tested best are rarely from one vendor. A cascade lets you route to any of 220+ LLMs at cost and pair it with the TTS voice you actually want.
You need transcript-native observability. Evaluation, moderation, analytics, and compliance review all operate on text. In a cascade, the transcript is the pipeline's ground truth, not a lossy afterthought.
You need mid-turn intervention. RAG injection, tool calls, PII redaction, and safety filters slot between STT and LLM or between LLM and TTS. End-to-end models offer no equivalent seam.
You need cost control at scale. Per-component pricing means you can cut LLM spend without touching voice quality, or change TTS tiers without retraining anything. At consumer volumes this is the difference between viable and non-viable unit economics.
You need independent upgrades. When a better LLM ships, you swap one config value. This is the classic trigger for migrating off OpenAI's speech-to-speech Realtime API: wanting a model upgrade the end-to-end product could not deliver.

Decision checklist

Work through these in order; the first strong "yes" usually decides it.

Does the product depend on hearing how the user speaks (emotion, pronunciation, laughter)? If yes, end-to-end.
Do compliance, moderation, or evaluation requirements demand text-level inspection of every turn? If yes, cascaded.
Do you need a specific LLM (or the freedom to change it quarterly)? If yes, cascaded.
Is this a prototype where shipping this week beats optimizing anything? If yes, end-to-end, and revisit at scale.
Are unit economics tight at high volume? If yes, cascaded; per-component pricing gives you levers a bundled audio-token price does not.
Do you need a cloned or designed brand voice? If yes, cascaded; end-to-end models ship fixed voice sets.
Still undecided? Build the cascade behind a clean interface. Moving from cascaded to end-to-end later is a small lift; the reverse means rebuilding turn-taking, voices, and observability from scratch.

About Inworld AI

Inworld is a research lab and inference provider focused on realtime AI models for consumer-facing applications. We build first-party voice models (Realtime TTS and Realtime STT), serve optimized open-source LLMs on our own Realtime Inference engine, and expose them as modular APIs, alongside an LLM Router that routes to 220+ models and a Realtime API for full speech-to-text-to-LLM-to-speech pipelines. We focus on serving developers of realtime, high-volume conversational products across domains such as health, fitness, education, companions, social, and games, with an emphasis on quality, low latency, and low cost at scale.

Frequently asked questions

What is the difference between a cascaded and a speech-to-speech voice architecture?

A cascaded architecture chains three specialized models: speech-to-text transcribes the user, a language model generates a text reply, and text-to-speech renders it as audio. An end-to-end speech-to-speech architecture uses one multimodal model that consumes audio and emits audio directly, with no text boundary in between. Cascaded systems give you model choice, transcript-native observability, and per-component pricing. End-to-end models preserve paralinguistic nuance like tone and hesitation and simplify the stack to a single vendor and connection.

Is a cascaded voice pipeline slower than an end-to-end speech-to-speech model?

Not inherently. Latency is a sum of component budgets, and a well-engineered cascaded pipeline achieves lower latency than some end-to-end speech-to-speech models. The TTS stage is no longer the bottleneck: Inworld's Realtime TTS 1.5 Mini streams first audio at roughly 100ms median, and Realtime TTS-2 at sub-250ms P90 (inworld.ai/tts, July 7, 2026). In practice LLM time to first token dominates the voice-to-voice budget in both architectures.

When should I use an end-to-end speech-to-speech model?

Choose end-to-end when maximum expressivity transfer matters more than control: the model hears the user's actual audio, so tone, hesitation, laughter, and accent survive into the response instead of being flattened into a transcript. It is also the simpler choice when you want one vendor, one connection, and one bill, and when your use case does not require swapping the underlying LLM, inspecting text between stages, or tightly controlling per-component cost.

What is the Inworld Realtime API?

The Inworld Realtime API is a configurable cascaded pipeline: speech-to-text, a language model of your choice, and text-to-speech, orchestrated over a single stateful connection so you do not stitch three vendors together yourself. "Realtime" refers to system responsiveness, not to an end-to-end speech model. It is a separate product from the Inworld TTS API, which you can use standalone inside any pipeline or framework.

How much does each architecture cost per minute of conversation?

End-to-end models price by audio tokens or minutes: OpenAI gpt-realtime-2.1 lists $32 per 1M audio input tokens and $64 per 1M audio output tokens, and Google quotes Gemini 3.1 Flash Live at $0.005/min in and $0.018/min out (both vendor pricing pages, fetched July 7, 2026). Cascaded stacks price per component: Inworld STT is $0.15/hr, TTS-2 is $25 per 1M characters on demand, and LLM routing is at cost across 220+ models (inworld.ai/pricing, July 7, 2026).

Can I switch between the two architectures later?

Yes, but the migration cost is asymmetric. Moving from cascaded to end-to-end mostly means replacing your pipeline with one connection and re-tuning prompts. Moving from end-to-end to cascaded means rebuilding turn-taking, picking an STT, LLM, and TTS, and re-creating your agent's voice. Keeping your system prompt, tool definitions, and evaluation transcripts vendor-neutral from day one keeps either migration small.

Cascaded vs Speech-to-Speech: Choosing Your Voice Agent Architecture