NVIDIA Releases Audio Model That Labels Up to Eight Speakers in Live Conversations

A live transcript can lose the thread when two people talk at once. NVIDIA released Nemotron 3 Diarization on September 23 to identify when each voice speaks in live or recorded audio. The 100-million-parameter model handles up to eight speakers and overlapping speech. It labels voices; a separate speech-recognition model is needed to supply their words.

Keeping a voice’s label through interruptions

Speaker diarization marks when each distinct voice is active. Nemotron assigns channels in the order voices first appear and estimates activity for each channel, so it can mark more than one speaker at a time. For live audio, a cache carries speaker information between chunks while a queue supplies recent context. That helps preserve a label after a pause or interruption without a separate voice-grouping stage.

The same model offers recommended input buffers from 30.4 seconds down to 0.32 seconds. A shorter buffer lets it produce labels sooner, with less audio context to work from. Those figures measure buffering, not the delay a listener would experience: computation and any added speech recognition take more time. They also say nothing by themselves about whether the speaker labels will be right.

The speed choice shows up in NVIDIA’s tests

On the DIHARD III evaluation set, NVIDIA reports that Nemotron’s diarization error rate rose as the buffer shortened. NVIDIA’s earlier Streaming Sortformer scored 19.09% with the longer buffer and 19.85% with the shorter one, compared with Nemotron’s lower rates shown below. These are company-reported speaker-labeling results, not tests of a finished transcription service.

12.73%30.4-second buffer

NVIDIA reports a 12.73% error rate with its longest recommended input buffer.

13.55%0.32-second buffer

NVIDIA reports a 13.55% error rate with its shortest recommended input buffer.

Baseten adds the words

Alongside the release, Baseten announced three serving presets: recorded-audio labeling, live labeling and live transcription with speaker labels. Each runs on one RTX PRO 6000 GPU. The transcription preset pairs Nemotron with NVIDIA’s Parakeet V2 speech-recognition model and returns speaker-tagged words, timings and overlap flags. It supplies the word-recognition step that the diarization model does not perform alone.

Baseten reports capacity for more than 500 simultaneous hour-long labeling streams at the 1.04-second buffer, versus 200 at 0.32 seconds. It reports 190 streams when transcription is added. Those figures describe Baseten’s setup, where a faster buffer setting or an added task changes how many streams one GPU can serve.

Available now, with a whole-system test ahead

The named Nemotron 3 Diarization repository is available on Hugging Face under the OpenMDW License Agreement, version 1.1; NVIDIA’s model card describes that model as ready for commercial or non-commercial use. For a live transcript, the practical question is how accurately a complete setup assigns words to speakers once recognition, processing and serving delays are included.