BroadRouter

Arabic ASR word error rate (WER) explained

By the BroadNet Inference team · 5 min read · Published

Word error rate (WER) is the number of substituted, deleted and inserted words divided by the words in the reference transcript. Lower is better. Arabic WER is hard to compare because diacritics, alef and ta marbuta spelling and dialects inflate it. BroadNet reports 3.8 WER on FLEURS-ar and 23.4 on dialect speech for BroadNet Transcribe Arabic.

Word error rate (WER) is the number of substituted, deleted and inserted words divided by the number of words in the correct transcript. Lower is better. Arabic WER is easy to inflate: the same correct transcript can be written several ways, and a naive comparison counts every spelling choice, diacritic and dialect spelling as a mistake.

This post explains what WER measures, why Arabic makes it slippery and how to compute it so your numbers mean something. To build the test set itself, see Arabic dialect speech recognition; to choose a provider, see the Arabic speech-to-text API buyer’s guide.

What is word error rate?

Word error rate compares a machine transcript (the hypothesis) with a correct human transcript (the reference). It finds the minimum number of word edits that turn one into the other:

  • S, substitutions: a word replaced by a different word.
  • D, deletions: a reference word missing from the hypothesis.
  • I, insertions: an extra word in the hypothesis.

Then:

WER = (S + D + I) / N, where N is the number of words in the reference.

A WER of 0.10 (often written as 10) means about one error per ten reference words. Lower is better. Because insertions count, WER can go above 100% for a transcript full of invented words.

Character error rate (CER) is the same formula over characters. It is useful for Arabic, where a single wrong letter in a prefix makes the whole word count as wrong.

Why is Arabic WER hard to compare?

1. Diacritics

Arabic short vowels are written as optional marks (tashkeel) above and below letters: fatha, damma, kasra, sukun, shadda and tanween. Everyday text leaves them out. If your reference has “مَدْرَسَةٌ” and the model writes “مدرسة”, a strict comparison counts a substitution, even though any Arabic reader sees the same word.

2. Spelling variants

Several letters have common, accepted variants:

Variation Example Typical normalisation
Hamza on alef أحمد / احمد, إلى / الى Map أ إ آ to ا
Ta marbuta vs ha مدرسة / مدرسه Map ة to ه
Alef maqsura vs ya على / علي (in loose writing) Map ى to ي
Tanween with alef صباحاً / صباحا Strip the diacritic
Tatweel (stretching) جـمـيـل / جميل Remove ـ

3. Clitics and word boundaries

Arabic attaches conjunctions, prepositions and pronouns to words: “وبمدرستهم” is “and in their school” in one written word. If the model writes “و بمدرستهم” with a space, the word count changes and WER punishes it twice. Some evaluations tokenise words into morphemes first; most do not. Know which you are reading.

4. Dialects

There is no standard spelling for dialect speech, unlike Modern Standard Arabic. An Egyptian speaker saying “I want” might be transcribed “عايز” or “عاوز”; a Gulf speaker’s “what” might be “وش” or “ويش”. Some transcribers also “correct” dialect into MSA. Each of these is a WER error even when the meaning is perfect. That is a large part of why dialect WER is so much higher than read-speech WER.

5. Numbers and code-switching

Speakers often switch into English mid-sentence (code-switching). Is it “٢٥”, “25” or “خمسة وعشرين”? Is an English brand name written in Latin script or in Arabic letters? Decide a convention, apply it to the reference, and normalise the hypothesis to it.

How much does normalisation change the score?

Reference: ذهبت إلى المدرسة صباحاً (“I went to school in the morning”)

Hypothesis: ذهبت الى المدرسه صباحا

Without normalisation, three of four words differ (إلى/الى, المدرسة/المدرسه, صباحاً/صباحا): WER 75%. After normalising both texts, they are identical: WER 0%. Same transcript, same quality, and the number moved from terrible to perfect. This is why a WER figure without its normalisation rules tells you little.

How do you compute Arabic WER in Python?

import re
import jiwer  # pip install jiwer

DIACRITICS = re.compile(r"[ً-ْٰ]")   # tanween, harakat, shadda, sukun, dagger alef
PUNCT = re.compile(r"[^\w\s]")

def normalise_ar(text: str) -> str:
    text = DIACRITICS.sub("", text)
    text = text.replace("ـ", "")                # tatweel
    text = re.sub("[أإآٱ]", "ا", text)
    text = text.replace("ى", "ي").replace("ة", "ه")
    text = text.translate(str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789"))
    text = PUNCT.sub(" ", text)
    return " ".join(text.split())

refs = ["ذهبت إلى المدرسة صباحاً"]
hyps = ["ذهبت الى المدرسه صباحا"]

print("raw WER:", jiwer.wer(refs, hyps))
print("normalised WER:", jiwer.wer([normalise_ar(r) for r in refs], [normalise_ar(h) for h in hyps]))
print("normalised CER:", jiwer.cer([normalise_ar(r) for r in refs], [normalise_ar(h) for h in hyps]))

Apply exactly the same function to references and hypotheses, for every model you compare. Publish it next to your numbers.

What WER does BroadNet report for Arabic?

BroadNet publishes lab measurements for BroadNet Transcribe Arabic in its model catalogue:

Test set What it is WER reported by BroadNet
FLEURS-ar Clean read sentences 3.8
MASC-ar Mixed real-world Arabic speech 21.1
BroadNet dialect set Dialect speech 23.4

BroadNet also reports the model at #2 of 37 (its September 2026 figure; the live board listed it #4 of 39 on 15 November 2026) on the Open Universal Arabic ASR Leaderboard, and #1 among models whose licences permit serving through an API. These are BroadNet’s figures, not independent audits. The leaderboard applies one evaluation setup to every model, which makes its ranks more comparable than numbers from separate vendor pages.

Read the spread, not just the best number. The jump from 3.8 to 23.4 is the gap between read speech and spontaneous dialect for one model, so test every candidate on both kinds of audio. Your own audio, whether call-centre lines, WhatsApp voice notes or meetings, will land somewhere in that range, and only a test tells you where.

How do you measure WER on your own audio?

  1. Sample by condition. Group clips by dialect and channel (phone, voice note, studio). The dialect test-set method covers sample sizes and spelling conventions.
  2. Transcribe with each candidate. For BroadNet, send language=ar; without it, Arabic can come back transliterated into Latin letters. The Arabic speech-to-text guide has the code, and the $5 starting credit covers about 1,190 minutes of audio.
  3. Normalise and score. Use the function above, report WER and CER per group.
  4. Read the errors. Sort substitutions by frequency. If the top errors are names, amounts or product codes, the business impact is bigger than the score suggests.

What should you measure besides WER?

WER treats every word equally, but your product does not. A support bot cares about intent; an invoicing workflow cares about numbers; a compliance team cares about specific phrases. Add a task metric next to WER: intent accuracy after an LLM reads the transcript, or exact match on extracted amounts. A model with slightly higher WER can still win if its errors fall on words that do not matter.

For the cost side, put your monthly audio hours into the free speech-to-text cost calculator, and see how the Arabic and 14-language models differ in BroadNet Transcribe vs Transcribe Arabic.

Questions

What is a good Arabic WER?

On clean read speech, single digits are achievable; BroadNet reports 3.8 on FLEURS-ar. On real-world and dialect audio, expect much higher numbers; BroadNet reports 21.1 on MASC-ar and 23.4 on its dialect set. Compare models on your own audio with the same normalisation.

Should I strip diacritics before computing Arabic WER?

Usually yes. Most Arabic is written without short-vowel marks, and models add them inconsistently. Strip them from both reference and hypothesis unless your product needs vowelled text.

What is CER and when should I use it?

Character error rate is the same calculation over characters instead of words. It is less harsh on Arabic, where one wrong prefix letter makes a whole word wrong, and is useful alongside WER.

Can WER be over 100%?

Yes. Insertions count as errors, so a model that invents many extra words can make more errors than there are words in the reference.

How do I calculate WER for Arabic in Python?

Install jiwer, normalise both the reference and the model output with the same function (strip diacritics and tatweel, unify alef, yaa and ta marbuta, convert Eastern Arabic digits), then call jiwer.wer(references, hypotheses). Report the normalisation next to the score.

Why is Arabic WER higher than English WER?

Partly real difficulty (dialects, clitics, code-switching) and partly scoring: Arabic has accepted spelling variants and optional diacritics, and a strict comparison counts each one as an error. Normalising both texts removes the scoring part.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity