08.28.2026
Introducing the Inworld TTS Open Evaluation Toolkit
An open customizable framework for reproducible, inspectable TTS evaluation.
Summarize with:
Aleksey Tikhonov, Head of ML/AI Research
We are open-sourcing the Inworld TTS Open Evaluation Toolkit. It samples TTS systems, evaluates the audio offline, and builds comparable reports. Each run stores its configuration beside the per-sample measurements. Users can trace a score to the source text and audio, inspect the ASR transcript, and check the applied thresholds. The first release includes common metrics and a specially crafted compact English dialogue stress set.
Motivation
TTS evaluations can be hard. Two teams can run the same TTS model on the same dataset and get different WER numbers: the choice of an ASR model and text normalization matter. Voice choice and synthesis settings do too. So does the decision to include or exclude a failed sample. We saw it a lot in practice. And once we move from basic metrics to subjective ones, like naturalness or expressiveness, things become even worse.
Modern systems mostly handle simple prompts well. Implementation details and failures on harder cases therefore have more influence on the result. A mean score without its protocol is rarely enough to reproduce the claim. Our goal is to provide an extensible and customizable tool for trustful and reproducible evaluations.
What we are releasing
The Inworld TTS Open Evaluation Toolkit keeps a run in one inspectable directory. The workflow is simple:
- Sample audio with fixed models and voices, recording the synthesis settings. Built-in adapters cover Inworld, ElevenLabs and Hume; any other system can provide audio through a manifest.
- Evaluate the clips offline with local ASR and explicit normalization. Compute audio metrics, then apply configurable thresholds.
- Report both the aggregate and the samples behind it.
The HTML report has no CDN or hosted runtime. Cross-run reports provide metrics by semantic groups (accuracy-related, subjective ones, failure modes) with confidence intervals and model-health summaries.
If a sample violates a threshold, the detailed version of the report places the expected text beside the ASR transcript. It also includes the audio when available.
The measurement cache is content-addressed. If the decoded audio and source text still match the metric configuration, the toolkit reuses its ASR and audio measurements. Thresholds and reports are still rebuilt, which makes tuning a decision rule much cheaper than transcribing the same batch again.
Metrics
The initial metric set covers several common things plus different failure surfaces:
- word- and character-level error rates, including insertion/deletion/substitution breakdowns;
- clipping and silence ratios, plus loudness and tail-click signals;
- duration and speaking-rate measurements;
- optional NISQAv2 dimensions and ECAPA speaker similarity;
- vowel-prolongation detection.
They should not be collapsed into one universal ranking. WER is also affected by mistakes made by the ASR evaluator. Low WER says little about whether a voice sounds natural.
We also provide lightweight arousal and expressiveness proxies. We discussed this split in Beyond Quality: Emotionality and Expressiveness. For a more direct look at those acoustic properties, the open-source Voice Lens and its browser demo calculate arousal and expressiveness locally from pitch/loudness/rhythm features. Uploaded audio stays in the browser.
Dialogue stress set
The repository includes a small English dialogue dataset built to surface failures quickly. Its 100 prompts include fragments and interruptions, along with emphasis markup. Repeated words, nested quotations and long dramatic lines make the small run more demanding than a list of simple benchmark sentences.
It is intentionally a stress set, not a representative sample of all speech. A deployment decision still needs production prompts in every relevant language. It also needs the voices the product will actually use. The bundled set is useful for finding obvious trouble before paying for the larger run.
Open by design
Even if reproducibility does not guarantee a correct evaluation, it makes the assumptions inspectable. We want teams to be able to rerun this framework and control its defaults. They can extend it with new metrics or datasets when the evaluation calls for them. The repository also includes agent instructions covering installation and the first end-to-end report from the command line.
Code and examples are in the repository README, together with the setup notes.