Introducing Voice Isolator
Remove background noise while keeping speech, laughter and breaths. Two 48 kHz models, for offline processing and for realtime streaming, with code and weights under Apache 2.0.View on GitHub ↗Hugging Face ↗
Today we're releasing our first open-source model: Voice Isolator. It keeps the voice as it was recorded, with its timbre, breaths and level, and removes background chatter, music and transients. It works at 48 kHz and ships as two checkpoints: an offline model for the best quality on existing recordings, and a realtime model that streams with 20 ms of look-ahead for live applications.
01Purpose
Removing traffic, music or someone talking in the background is half the job. The other half is keeping the person you want to hear. Most denoisers treat a laugh, a breath or a sigh as noise and take it out. Voice Isolator is trained to keep them.
Listen to it on four noisy clips. Switch isolation off to hear what the microphone captured.
Car
Vacuum cleaner
Blender
Rock music
In a live conversation, those sounds tell a voice agent a lot: a breath before someone speaks, a laugh in response, or an "hmm" while they think. Keeping them helps the agent decide when to respond. In a recorded interview or a voice sample, they keep the character of the original voice.
02Evals
We built VIBE (Voice Isolation Benchmark and Evals) to measure how well a model preserves the original voice while removing everything else. It is a multilingual, multi-condition set of real utterances, with and without added noise, and we use it to score open-source models against each other.
- Other open-source denoisers
Our models performed best among the open-source denoisers we were able to test.
VIBE is a private benchmark for now. If you'd like to submit a model, or help improve Voice Isolator or the benchmark itself, reach out and we'll be happy to collaborate.
03How to
Start with a recording you know well. Install the package, then run it on the file using the commands below. The weights download from Hugging Face on first use. The model runs on CUDA, Apple silicon and CPUs.
pip install inworld-voice-isolation
# the voice goes to clean.wav, what was removed to clean_background.wav
voice-isolate noisy.wav clean.wav
# the realtime model, as a stream
voice-isolate noisy.wav clean.wav --mode streaming --lookahead 2from inworld import voice_isolation
live = voice_isolation.VoiceIsolator("inworld/voice-isolator",
mode="streaming", lookahead_frames=2)
session = live.stream() # 48 kHz mono, in pieces of any size
for piece in microphone:
play(session.push(piece)) # the samples that are final
play(session.flush())To hear it for yourself, run the repository's interactive demo on your laptop. It has a tab for recorded files and another that sends microphone audio through the realtime model.
04Next
Voice Isolator currently only does denoising. We trained it this way on purpose, so that it introduces no bias into downstream training tasks or into realtime speech-to-text models, which can be sensitive to changes in the waveform.
But denoising alone won't fix a low-quality microphone or missing band detail. That is restoration. We're looking at it, and we may have something to share.
We'd like to hear how Voice Isolator works in your application. If you have an idea for making it better, or you found something that doesn't work well, create an issue or ask your agent to open a pull request, and we'll respond. Let's make it better together.