BroadRouter
Get started

Speaker diarization

Updated

Speaker diarization is the task of dividing an audio recording into segments by speaker, answering "who spoke when". It labels each stretch of speech with an anonymous speaker ID such as SPEAKER_00, and is usually combined with a transcript to show which person said each line.

Also called: speaker diarisation, speaker labels, who spoke when, speaker separation, speaker segmentation

Diarization turns one block of transcript into a conversation: agent, then customer, then agent again.

How it works

A diarization pipeline usually runs three stages:

  1. Find speech. Voice activity detection marks where anyone is talking.
  2. Embed short windows. A speaker-embedding model turns each slice into a vector that captures the voice.
  3. Cluster. Slices with similar vectors get the same label, and overlapping speech is handled where the model supports it.

The open pyannote speaker-diarization-3.1 pipeline on Hugging Face takes mono audio at 16 kHz (it downmixes stereo and resamples other rates) and writes results in RTTM, a plain-text format with one line per speaker turn. An RTTM line gives the file, channel, start time, duration and speaker label:

SPEAKER call42 1 0.00 4.21 <NA> <NA> SPEAKER_00 <NA> <NA>
SPEAKER call42 1 4.35 2.80 <NA> <NA> SPEAKER_01 <NA> <NA>

Its model card reports diarization error rate (DER) per benchmark, from 7.8% on REPERE to 50.0% on AVA-AVD, with 11.3% on VoxConverse. The spread shows how much results depend on the audio: meetings, TV and phone calls behave differently.

Some hosted APIs do this for you. OpenAI’s speech-to-text guide documents a diarization model, gpt-4o-transcribe-diarize, with a diarized_json response whose segments carry speaker, start and end fields, and optional reference clips of 2 to 10 seconds for up to four known speakers. OpenAI lists that model as deprecated, shutting down on 26 February 2027, and its guide (checked 2026-11-15) documents speaker labels on no other model.

Why it matters

  • Call centers and interviews. Quality review, compliance and summaries all need to know who said what.
  • Downstream LLM work. A summary prompt with “Agent:” and “Customer:” turns gives better answers than one wall of text.
  • Cost. Diarization is extra compute, priced separately or bundled, depending on the provider. The speech-to-text API price index notes which providers charge extra for it.

On BroadNet Inference

BroadNet’s docs do not list diarization. The transcription endpoint returns {"text": "..."} only, with no speaker labels or timestamps. A practical pattern is to run an open diarizer such as pyannote locally, cut the audio at each speaker turn with ffmpeg, and send each clip to BroadNet Transcribe Arabic or BroadNet Transcribe with language set. If the two sides are on separate stereo channels, split the channels instead; that needs no model at all. See transcribing call center recordings for the wider workflow.

Questions

Does diarization tell me the speaker's name?

No. It returns anonymous labels like SPEAKER_00 and SPEAKER_01. Some APIs, such as OpenAI's gpt-4o-transcribe-diarize (deprecated, shutting down 26 February 2027), can match up to four speakers to short reference clips you supply.

Does BroadNet's transcription API do diarization?

BroadNet's docs do not list diarization. The endpoint returns the transcript as {"text": ...}, so run a separate diarization model if you need speaker turns.

How is diarization accuracy measured?

With diarization error rate (DER), which adds false alarm, missed speech and speaker confusion time, then divides by the total reference speech time. Lower is better.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity