BroadRouter

Add diacritics to Arabic text with an LLM: prompt, checks, DER

By the BroadNet Inference team · 8 min read · Published · Updated

Ask the model for full tashkeel and nothing else, then strip the diacritics from its reply and compare with your input. Any changed letter means reject and retry. Score a hand-checked sample with diacritic error rate (DER) and word error rate (WER), with and without case endings. Published results put GPT-4 close to, but behind, the best dedicated diacritizers.

Arabic is normally written without short vowels. A reader fills them in from context; software often can’t. Restoring them, called diacritization or tashkeel, turns كتب into كَتَبَ (he wrote), كُتُبٌ (books) or كُتِبَ (it was written), depending on the sentence. This post shows how to do it with an LLM through an OpenAI-compatible API, how to stop the model from quietly changing your text, and how to measure the result against a dedicated diacritizer.

When do you need diacritics?

  • Text-to-speech. A voice needs the vowels to pronounce a word. The CATT paper lists text-to-speech and machine translation as applications that diacritics improve. BroadNet Inference has no text-to-speech endpoint, so in a TTS pipeline the diacritized text goes to a separate TTS service.
  • Learners and children’s material. Readers who are still learning rely on the marks.
  • Ambiguous names and terms. Proper nouns, rare words and technical terms where a wrong vowel changes the meaning.
  • Religious and classical text. Here the marks are part of the text, and an automatic tool should only fill gaps under human review.

You usually don’t want diacritics for search or retrieval. There you remove them, as in Arabic text normalization for search. The free Arabic text normalizer does that removal in the browser. Diacritization is the opposite operation, and much harder: removing marks is a lookup, while adding them needs grammar and context.

How good are LLMs at diacritization?

Two published numbers, read with care:

System CATT benchmark, DER with case endings DER without WER with case endings WER without
CATT encoder-decoder 8.624% 6.989% 34.191% 28.477%
CATT encoder-only 8.762% 7.088% 35.508% 29.714%
GPT-4 (GPT-4-turbo per the paper’s text) 9.515% 8.113% 38.311% 33.505%
Command R+ 13.169% 11.329% 48.518% 44.158%
Shakkala 13.494% 10.386% 50.387% 40.643%
Mishkal 16.482% 10.796% 60.844% 40.215%

Source: Table 4 of the CATT paper (2024). Lower is better. The benchmark is 742 news sentences scraped in 2023, diacritized by two native-speaker experts and checked by a third.

On the paper’s second test set, WikiNews (400 sentences), GPT-4 scored far better than everything else: 0.551% DER. The authors mark that row and suggest the model may have seen WikiNews in training. That is the general warning for LLM benchmarks: public test text may be in the training data. The CATT authors raise the same concern about Tashkeela, whose books contain much Quran and hadith quotation.

A separate 2024 paper by Skiredj and Berrada reports GPT-4 at about 20% DER and 30% WER on the Fadel et al. benchmark, without the detail we would need to compare it with the table above.

So GPT-4-class models can come close to a dedicated diacritizer, and they can also trail it. Both papers predate the open models you can call today. Measure on your own text.

Which test sets exist?

Dataset What it is
Tashkeela 75,629,921 fully vocalized words from 97 books; 98.85% classical Arabic from the Shamela library, 1.15% modern Standard Arabic
Fadel et al. benchmark About 2.3M words extracted from Tashkeela; 50,000 training, 2,500 validation and 2,500 test lines
CATT benchmark 742 manually diacritized news sentences, released with the CATT models
WikiNews 400 manually diacritized MSA news sentences

Tashkeela and the Fadel set are mostly classical. If your text is modern business or news Arabic, the CATT and WikiNews sets are closer, and a sample of your own text is closest.

What prompt keeps the letters intact?

The model’s job is to add marks and touch nothing else. LLMs like to help, and they will fix الى to إلى, change a final ى to ي or add a missing hamza. Each of those may be correct Arabic, but it is a spelling correction, not diacritization. It changes your text without telling you.

So the prompt states the rule, and code enforces it: strip the diacritics from the reply and compare with the stripped input. If one letter, space or punctuation mark differs, the reply is rejected.

import difflib, os, re, unicodedata
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

MARKS = re.compile("[ً-ْٰ]")  # tanween, harakat, shadda, sukun, dagger alef
SYSTEM = """Add full Arabic diacritics (tashkeel) to the user's text, including case endings.
Reply with the diacritized text only. Do not add, remove, reorder or replace any letter,
space, digit or punctuation mark. Do not fix spelling. Keep any diacritics already present."""

def skeleton(s: str) -> str:
    """The text with every diacritic removed: what the model is not allowed to change."""
    return MARKS.sub("", unicodedata.normalize("NFC", s))

def diacritize(text: str, model: str = "broadnet/qwen3.8-27b", tries: int = 2) -> str | None:
    for attempt in range(tries):
        r = client.chat.completions.create(
            model=model, max_tokens=4000, temperature=0,
            messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}])
        out = unicodedata.normalize("NFC", (r.choices[0].message.content or "").strip())
        if skeleton(out) == skeleton(text):
            return out
        diff = [d for d in difflib.ndiff(skeleton(text).split(), skeleton(out).split()) if d[0] in "+-"]
        print(f"attempt {attempt + 1}: letters changed {diff}")
    return None  # keep the undiacritized text; never ship a changed sentence

for line in ["ذهب الطالب الى المدرسة", "قرأ محمد الكتاب"]:
    print(diacritize(line))

We ran this against a local mock server, not the live API. Its canned reply for the first sentence “fixed” الى to إلى on both attempts, and the reply for the second was clean. The output was:

attempt 1: letters changed ['- الى', '+ إلى']
attempt 2: letters changed ['- الى', '+ إلى']
None
قَرَأَ مُحَمَّدٌ الْكِتَابَ

Notes on the code:

  • Unicode NFC first. A hamza can arrive as one precomposed character (أ) or as a letter plus a combining mark. NFC makes both sides the same before comparing. The normalization post covers this trap in detail.
  • max_tokens=4000. BroadNet’s docs say DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering, and the reasoning counts toward max_tokens. Too small a limit returns an empty reply, which the check rejects. Leave room, or set reasoning_effort to low; see empty responses from reasoning models and the reasoning token budget calculator.
  • The check does not test “keep existing diacritics”. If your input is partly diacritized, compare those marks too.
  • Spelling is a separate decision. If you want الى corrected, run a correction step first, then diacritize the corrected text. Mixing the two makes both impossible to check.
  • Chunk long text. Send a sentence or a paragraph per request. CATT itself caps input at 1,024 characters and splits long sentences at punctuation; the same habit keeps LLM replies short and retries cheap.

For general advice on writing Arabic prompts, including when to tell a model not to add diacritics, see Arabic prompt engineering.

How do you score diacritization: DER and WER?

Diacritic error rate (DER) is the share of letters whose diacritics differ from the reference. Word error rate (WER) here is the share of words with at least one wrong diacritic. That is not the speech-recognition WER in our ASR post; it counts words with a wrong mark, not inserted or deleted words.

Two settings change the numbers a lot:

  • With or without case endings. The last letter of a word carries its grammatical case (i’rab), which depends on syntax. Papers report both settings, because case endings cause many of the errors.
  • Counting letters with no diacritic. Some scripts count a bare letter (such as the alef in الطالب) as a correct “no diacritic” prediction; some skip it. The Fadel scoring script has a flag for this.

Published scripts differ in these choices, so only compare numbers produced by the same script. This one is short enough to read:

import re, unicodedata

MARK = re.compile("[ً-ْٰ]")
LETTER = re.compile("[ء-غف-ي]")

def letters(word: str) -> list[tuple[str, str]]:
    """(letter, its diacritics in a fixed order) for each Arabic letter in a word."""
    out = []
    for ch in unicodedata.normalize("NFC", word):
        if LETTER.match(ch):
            out.append([ch, ""])
        elif MARK.match(ch) and out:
            out[-1][1] += ch
    return [(c, "".join(sorted(d))) for c, d in out]

def der_wer(gold: list[str], pred: list[str], case_ending=True, count_bare=True) -> dict:
    """DER: share of scored letters with the wrong diacritics. WER: share of words with any such letter.
    case_ending=False skips each word's last letter. count_bare=False skips letters with no gold diacritic."""
    errs = total = werr = words = 0
    for g_line, p_line in zip(gold, pred):
        for g, p in zip(g_line.split(), p_line.split()):
            G, P = letters(g), letters(p)
            if [c for c, _ in G] != [c for c, _ in P]:
                raise ValueError(f"letters differ: {g} / {p}")  # score only same-skeleton output
            scored = [(gd, pd) for i, ((_, gd), (_, pd)) in enumerate(zip(G, P))
                      if (case_ending or i < len(G) - 1) and (count_bare or gd)]
            if not scored:
                continue
            bad = sum(gd != pd for gd, pd in scored)
            errs += bad; total += len(scored); werr += bad > 0; words += 1
    return {"DER": round(100 * errs / total, 2), "WER": round(100 * werr / words, 2)}

gold = ["ذَهَبَ الطَّالِبُ إِلَى المَدْرَسَةِ", "قَرَأَ مُحَمَّدٌ الكِتَابَ"]
pred = ["ذَهَبَ الطَّالِبَ إِلَى المَدْرِسَةِ", "قَرَأَ مُحَمَّدٌ الكِتَابُ"]
for ce in (True, False):
    print("with case endings" if ce else "without case endings", der_wer(gold, pred, case_ending=ce))

The toy prediction has two wrong case endings and one wrong internal vowel across 32 letters and 7 words. It prints DER 9.38 and WER 42.86 with case endings, and DER 4.0 and WER 14.29 without. Note how WER magnifies a few errors: one wrong mark spoils a whole word.

The script refuses to score a word whose letters changed. That is deliberate. A diacritizer that edits letters has failed the task, and its DER would be meaningless.

As a second check we ran Mishkal 0.4.1, a rule-based diacritizer, on one 16-word sentence we diacritized by hand. All four of its errors were case endings, such as مُحَمَّدُ for مُحَمَّدٌ. It scored DER 5.88 with case endings and 0.0 without. One sentence proves nothing about Mishkal; it shows why you report both settings.

To build a test set, take 100 to 200 sentences of your own text, have a native speaker diacritize them, strip the marks, and run each candidate. Keep the human version as gold.

When does a dedicated diacritizer beat an LLM?

Choose When
A dedicated model (CATT, libtashkeel, Mishkal) High volume, fixed text type, strict latency, offline or on-device use, or a need for letter-for-letter output by design
An LLM through an API Low or uneven volume, mixed or messy input, a team without ML ops, or a pipeline that already calls an LLM for other steps

Dedicated models predict a mark per letter, so they can’t rewrite your text. CATT is Apache 2.0 and installs with pip install catt-tashkeel; its README recommends the encoder-only model for speed and the encoder-decoder for accuracy. libtashkeel is MIT and trained mainly on MSA. Mishkal is rule-based and GPL, which matters if you ship it inside a product.

An LLM is the faster start and copes with input a trained model never saw: dialect, mixed Arabic and English, odd punctuation. It needs the skeleton check above on every reply, and it costs per token.

What does LLM diacritization cost?

Diacritics are extra characters, and tokenizers split them into extra tokens. We counted the 2,500-line Fadel test set (125,098 words) with each model’s own tokenizer, bare text against fully diacritized text:

Tokenizer Bare text Diacritized Ratio
Qwen3.8 27B 196,987 407,144 2.07×
DeepSeek V4.1 Flash 222,555 540,454 2.43×
Gemma 4 VL 26B 208,769 508,758 2.44×
GLM 5.3 239,753 770,361 3.21×

You pay input tokens for the bare text and output tokens for the diacritized reply, plus any reasoning. Output is the expensive side: $2.17 per million tokens on Qwen3.8 27B, $0.52 on DeepSeek V4.1 Flash, $3.46 on GLM 5.3 and $0.22 on Gemma 4 VL 26B, from BroadNet’s pricing page. Retries after a failed check add more. Paste your own text into the Arabic token counter to check the ratio on your data, and see tokens per word in Arabic and English for why Arabic costs more tokens to begin with.

For a one-off job over a large archive, the DeepSeek V4.1 Flash plans replace the meter with a flat price per session ($15 a week or $1 an hour); each session runs one request at a time, so plan the run as in batch processing without a batch API.

Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference serves the four chat models above through one OpenAI-compatible endpoint, so the code runs with the OpenAI Python SDK once the base URL and key are set. Gemma 4 VL 26B runs on the $5 starting credit; DeepSeek, GLM and Qwen need a top-up. BroadNet has no diacritization endpoint, publishes no diacritization benchmark of its own and has no text-to-speech. Which of its models diacritizes your text best is unmeasured: score them on your own sample before choosing. See the API reference for request details.

Questions

How do I add diacritics to Arabic text in Python?

Two routes. Install a dedicated diacritizer such as catt-tashkeel (Apache 2.0) or mishkal (GPL) and call it locally, or send the text to an LLM with a prompt that asks for tashkeel only. With an LLM, always check that the reply minus its diacritics equals your input.

Is ChatGPT or GPT-4 good at Arabic diacritization?

Close to the best dedicated systems on some tests, behind them on others. The CATT paper measured GPT-4 at 9.515% DER with case endings on its news benchmark, against 8.624% for its own model. Another 2024 paper reports about 20% DER for GPT-4 on the Fadel benchmark. Test on your own text.

What is the difference between DER and WER for diacritization?

DER is the share of letters whose diacritics are wrong. WER is the share of words with at least one wrong diacritic, so it is always higher. Both are reported with case endings (the last letter of each word) and without them.

Why does my LLM change letters when adding tashkeel?

It also corrects spelling: الى becomes إلى, a ى becomes ي, a missing hamza appears. That may be right, but it is a correction, not diacritization, and it breaks letter-by-letter scoring. Ask it not to, check every reply, and run spelling correction as a separate step if you want it.

Can BroadNet read the diacritized text aloud?

No. BroadNet Inference has no text-to-speech endpoint. It offers chat models and speech-to-text. If diacritization is the front end of a TTS pipeline, send the result to a separate TTS service.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity