NVIDIA unveils eight-speaker diarization model with minimum recommended buffering latency of 320 milliseconds

NVIDIA introduced Nemotron 3 Diarization on September 23: a 100-million-parameter speaker diarization model that increases the maximum number of supported speakers from four in the previous generation to eight, while supporting both recorded and streaming audio. The model outputs activity timestamps for anonymous speakers, which can be used to label who spoke when in a transcript. It does not generate text or identify speakers by their real-world identities. [Official technical overview](https://huggingface.co/blog/nvidia/nemotron-diarization)

Its streaming design carries forward Streaming Sortformer, assigning speaker channels in the order speakers first speak. The AOSC cache described in the existing paper stores acoustic representations of earlier speech and selects segments to retain based on prediction scores; a FIFO queue supplies recent context. This mechanism addresses a central challenge: maintaining consistent labels across audio segments when the same person resumes speaking after a pause, without having to reorder channels each time. [Architecture paper](https://arxiv.org/abs/2507.18446)

The new model’s minimum recommended buffering setting is 0.32 seconds, with additional configurations of 0.64, 1.04, and 30.4 seconds. These figures are calculated from the current audio chunk plus right context and exclude model computation, networking, speech recognition, and application processing. Shorter buffering leaves less context available for decisions, so 320 milliseconds cannot be treated as the time until a user sees captions. [Latency settings](https://huggingface.co/blog/nvidia/nemotron-diarization)

The official report states that, with 1.04 seconds of buffering, the relative improvement in diarization error rate averaged 41% across eight evaluation conditions. This is an unweighted average of the relative improvements under each condition, rather than an error rate calculated by pooling all recordings. It also does not imply that every type of Chinese-language meeting will see the same improvement. [Evaluation methodology](https://huggingface.co/blog/nvidia/nemotron-diarization)

To reproduce the results, the NeMo evaluation script requires a data manifest linked to timestamp annotation files and provides settings for the scoring collar, whether overlapping speech is included, and post-processing. The script documentation also notes that reproducing the model card’s results requires the corresponding post-processing; using the default binarized output directly may not yield the same scores. [Evaluation script](https://github.com/NVIDIA-NeMo/Speech/blob/main/examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py)

Two simultaneously active speaker channels indicate only when speech overlaps. Assigning a recognized word to one of those speakers still requires separate alignment and ambiguity resolution. Deployment teams should therefore next test speaker attribution errors in complete transcripts and end-to-end latency using recordings featuring Chinese speech, code-switching, long pauses, and interruptions by multiple speakers. The official model card could not be accessed during this review, so the conditions for downloading the weights remain unconfirmed. The performance results in the announcement have also not been independently reproduced as part of this review.