Published 05.28.2026
What Is Controllable TTS? Definition, Examples, and How It Works
Summarize with:
TL;DR: Controllable TTS lets developers direct how generated speech is performed without changing the words or retraining the voice. Inworld Realtime TTS-2 uses bracketed natural-language instructions, adjustable pauses, non-verbal sounds, and stable voice identities. Production testing should measure instruction accuracy, consistency, latency, and failures on real application text.
Controllable text-to-speech separates three decisions that older speech systems often combine: what is said, who says it, and how the line is performed. A developer supplies the script, selects or creates a voice, and adds delivery instructions for tone, speed, volume, pitch, pauses, or vocal style. The result is useful only when those controls work predictably inside a realtime application.
The public launch of Inworld Realtime TTS-2 makes this category concrete. TTS-2 accepts bracketed instructions and renders non-verbal cues such as laughter, breathing, throat clearing, sighing, coughing, and yawning as sounds rather than spoken stage directions. The same model supports streaming, voice design, cloning, and multilingual delivery, so controllability can be evaluated as part of the full speech pipeline.
What does controllable text-to-speech actually mean?
Controllable TTS is speech synthesis that accepts explicit direction over delivery while preserving the underlying script and speaker identity. Controls may govern emotion, speed, volume, pitch, articulation, pauses, or non-verbal sounds. The defining test is whether a developer can request a change and obtain a repeatable, measurable difference in the output.
This definition excludes a simple catalog of preset voices. Choosing between a warm narrator and a bright assistant changes the speaker profile, but it does not necessarily direct a specific line. Controllable systems expose request-level instructions, allowing the same voice to sound measured in one turn and urgent in the next. That matters in games, tutors, companions, and voice agents where the right performance depends on context generated at runtime.
Which speech attributes can developers control?
Useful control surfaces map instructions to audible properties: emotion, speed, volume, pitch, intonation, articulation, vocal style, pauses, and non-verbal expression. A model should expose only controls it can render reliably. Long feature lists have little value when instructions conflict, drift across retries, or alter the words being spoken.
- Tone and emotion: calm, reassuring, excited, hesitant, stern, or playful delivery.
- Timing: speaking speed, phrase-level pauses, and emphasis around key words.
- Acoustic character: pitch, volume, range, articulation, and vocal style.
- Non-verbal expression: laughter, breathing, throat clearing, sighing, coughing, and yawning.
- Identity consistency: keeping the same speaker recognizable as delivery and language change.
Inworld documents bracketed instructions within the text. A delivery instruction belongs at the start of the text, because mid-line placement of a delivery instruction is unreliable. Non-verbal cues are the exception: they can appear inline at the moment the sound should occur. The model's control surface should be tested with neutral scripts and emotionally loaded scripts because content and direction can conflict. A sad instruction paired with celebratory wording may produce unstable results rather than a useful blend. Teams should define which combinations are allowed before exposing free-form controls to users.
How does natural-language TTS steering work?
Natural-language steering places a short instruction beside the text to be synthesized. The model treats the instruction as performance direction rather than content to pronounce. This removes the need to train a new voice for every delivery style, but it also creates a prompt-design problem: directions must be specific, compatible, and tested against production text.
A request might open with
[calm, reassuring] before a support response or [excited] before a game event. Inline cues can add [laugh] or [sigh] at the moment they should occur. The application still needs guardrails. Instructions should come from a controlled vocabulary where consistency matters, while open-ended directions can be reserved for creative tools. Inworld's TTS documentation also distinguishes delivery controls from voice settings and audio-format choices.Keep the script, voice ID, audio format, and model version fixed when evaluating a steering instruction. Changing several variables at once makes the result impossible to attribute.
How do pauses and non-verbals change meaning?
Pauses and non-verbal sounds carry information that words do not. A breath can signal hesitation, a sigh can mark frustration, and a pause can change which phrase receives emphasis. These controls improve a scene only when they match the context. Overuse makes generated speech feel staged and can slow conversational turn-taking.
The cleanest evaluation uses one sentence across several versions. Generate a neutral baseline, then add one delivery instruction, one pause, or one cue at a time. Ask listeners what emotional intent they heard without showing the prompt. If raters cannot identify the requested difference, the control failed. If they hear a different intent or the cue distracts from the words, the control produced the wrong effect.
- Start with a neutral generation.
- Change one control and keep every other input fixed.
- Randomize playback order and hide the prompt.
- Record perceived intent, intelligibility, and preference.
- Repeat across short turns, long turns, languages, and peak traffic.
How should controllable TTS models be compared?
Compare controllable TTS models with blinded tests on the same scripts, voices, directions, and latency conditions. Separate control accuracy from general voice preference. A model can sound appealing yet ignore instructions, or follow instructions while damaging intelligibility. Production evaluation should report both behavior and tail latency under the concurrency the application expects.
The Artificial Analysis Speech Arena uses blind pairwise listening to rank overall preference. On September 2, 2026, Inworld Realtime TTS-2 ranked second with an Elo score of 1,250, while the model listed by Artificial Analysis as TTS-2 Flash research preview ranked sixth at 1,222. These external signals do not isolate instruction following or predict results on a product's own text. Use this ranking only as directional market evidence.
How do current TTS models compare externally?
External leaderboards measure overall listener preference, not steering accuracy, but they place controllable models in a market context. On the Artificial Analysis Speech Arena on September 2, 2026, Cartesia Sonic 3.6 ranked first, Inworld Realtime TTS-2 second, and the model it listed as TTS-2 Flash research preview sixth in blind pairwise listening tests.
The Artificial Analysis Speech Arena also listed SpeechifyAI Simba 3.2, Alibaba Qwen-Audio-3.0-TTS-Plus, ElevenLabs v3 Conversational, and Google Gemini 3.1 Flash TTS in the top ten. Hume AI, Deepgram, OpenAI, Microsoft Azure Speech, and Amazon Polly serve adjacent speech needs with different strengths. A leaderboard cannot decide which control surface fits an application, so teams must test instruction following independently with their own scripts.
Which product outcomes should controllability improve?
Controllability matters only if it improves a product outcome: clearer instruction, stronger engagement, fewer regenerations, better completion, or more consistent character behavior. Inworld customer results show that speech changes can affect usage and retention, but they do not isolate steering. Teams should design experiments that connect each control to a measurable behavior.
According to Inworld's Talkpal case study, a four-week A/B test reported 40% lower TTS cost, 7% higher feature usage, and 4% higher retention after a speech-model change. Wishroll's Status case study reported about 95% lower AI cost at more than 500,000 daily active users. These results support testing speech as a product variable, without claiming steering directly caused those gains.
- Tutoring: correction acceptance and lesson completion.
- Companions: session length and voluntary return rate.
- Games: dialogue completion and character consistency ratings.
- Voice agents: interruption recovery and task completion.
- Accessibility: comprehension and playback abandonment.
When is controllable TTS the wrong choice?
Controllable TTS is unnecessary when an application needs a fixed announcement voice, batch narration with extensive post-production, or deterministic prerecorded playback. It can also be risky when users may insert manipulative cues, clone voices without permission, or create conflicting directions the product cannot safely moderate.
Recorded audio remains more predictable for legal disclosures, emergency instructions, and prompts that never change. Creator-focused systems may fit long scripted narration where editing time matters more than realtime latency. A controllable model is strongest when lines are generated dynamically and delivery must adapt on each turn. The architecture should follow the interaction and its risk, not the novelty of the feature.
Voice cloning adds separate obligations. Teams need documented consent, usage rights, revocation procedures, and controls over who can invoke the voice. Inworld states that instant cloning can use five to 15 seconds of authorized audio, but a short technical requirement does not reduce the legal or ethical burden. The application remains responsible for the source recording and every permitted downstream use.
How should teams implement steering in production?
Production steering should begin with a bounded instruction library, versioned prompts, and a regression set drawn from real product text. Teams should log the model, voice, direction, latency, and output result for every test. Free-form instructions can expand later, once failure patterns and moderation rules are understood.
Start with the ten or 20 delivery intents the application uses most. Define one canonical instruction for each intent and test it across voices, languages, sentence lengths, and emotional contexts. Store audio examples that represent acceptable and unacceptable results. When a model version changes, rerun the same set before expanding traffic. This turns subjective listening into a release process with traceable decisions.
- Version instruction templates beside application code.
- Log the exact model and voice ID used for each output.
- Track P95 and P99 latency, not only averages.
- Review altered words and unwanted sounds as defects.
- Provide fallback delivery when a control fails.
Related Guides
- Inworld Realtime TTS-2 and TTS-2 Flash
- How to Evaluate TTS Models for Realtime Conversational AI
- Cross-Lingual Voice Cloning in 2026
- Best TTS API for Developers in 2026
- Voice AI for Consumer Apps: A 2026 Stack Guide
Key Takeaways
- Controllable TTS separates the script, speaker identity, and delivery instructions into independent production choices.
- Natural-language directions are useful only when their audible effects are repeatable, intelligible, and measurable.
- Delivery instructions belong at the start of the text; non-verbal cues can be placed inline where the sound should occur.
- Blind tests should score instruction accuracy separately from overall voice preference and latency.
- Voice cloning requires consent, rights, revocation controls, and documented limits regardless of sample length.
Frequently Asked Questions
What is controllable TTS?
Controllable TTS is text-to-speech that accepts explicit direction over delivery, such as tone, speed, volume, pitch, pauses, or non-verbal sounds. The developer keeps the script and speaker identity separate from those instructions. A controllable model should render requested changes reliably without altering the words or causing unacceptable latency.
How is TTS steering different from voice cloning?
Voice cloning determines who the synthetic speaker sounds like, while steering determines how that speaker performs a particular line. A product may clone or design a voice once, store its voice ID, and then apply different delivery instructions on every request. The two features solve different problems and are often used together.
Can controllable TTS work in realtime applications?
Yes, when the model streams audio quickly enough and the control instructions do not create unacceptable tail latency. Teams should measure time to first byte, first playable audio, P95 and P99 performance, and end-to-end turn latency under realistic concurrency. A strong offline sample does not prove that a model will sustain conversation timing.
How should teams test TTS emotion controls?
Use blinded pairwise listening tests on production scripts. Keep the model, voice, text, and audio format fixed, then change one delivery instruction at a time. Ask listeners to identify the perceived intent without seeing the prompt. Track instruction accuracy, intelligibility, identity stability, preference, and latency rather than relying on a single quality score.
When should developers avoid controllable TTS?
Avoid it when prerecorded audio is safer and more predictable, such as fixed legal notices or emergency instructions, or when batch narration will receive extensive human editing. Teams should also avoid open user-controlled steering until they can moderate manipulative cues, unauthorized voice use, conflicting instructions, and other foreseeable misuse.
Published by Inworld. Product capabilities and pricing were checked against Inworld documentation on September 2, 2026. Artificial Analysis leaderboard positions reflect the live Speech Arena on that date and may change. Product teams should validate control accuracy, latency, language quality, consent, and rights requirements against their own workloads.