BroadRouter

Arabic diacritics (tashkeel)

Updated

Arabic diacritics, or tashkeel, are small marks written above or below letters to show short vowels, doubled consonants and the absence of a vowel. They are optional: most Arabic text, including news, omits them, so readers and software infer the vowels from context.

Also called: tashkeel, tashkil, harakat, diacritization, Arabic diacritization, diacritics, vowel marks

The same three Arabic letters can mean “he wrote”, “books” or “it was written”. Diacritics are what tell them apart, and most text leaves them out.

How it works

The Unicode Standard calls these marks tashkil and says they “may be written as combining marks”, then adds: “In normal writing, however, these marks are omitted.” Harakat is the name for the short-vowel subset. Each mark is its own combining character that follows the letter it sits on. The main ones are in the range U+064B to U+0652:

Code point Name Role
U+064E FATHA short a
U+064F DAMMA short u
U+0650 KASRA short i
U+064B to U+064D FATHATAN, DAMMATAN, KASRATAN tanween (-an, -un, -in endings)
U+0651 SHADDA doubled consonant
U+0652 SUKUN no vowel

A worked example with the bare word كتب:

  • كَتَبَ (kataba), “he wrote”
  • كُتُب (kutub), “books”
  • كُتِبَ (kutiba), “it was written”

Without marks, all three are the same string. A reader picks the meaning from the sentence. Software has to do the same, which is the job of a diacritizer. Zerrouki and Balla (2017) built the Tashkeela corpus, about 75.6 million words, mostly Classical Arabic, because fully vowelled text is scarce and diacritizers need it for training. Fadel et al. (2019) report neural diacritization models that match or beat earlier work on a public benchmark, and show that diacritics can help machine translation.

Libraries treat these marks as one set. CAMeL Tools’ AR_DIAC_CHARSET holds U+064B to U+0652 plus U+0670 (superscript alef), and its dediac_ar function removes them.

Why it matters

  • Search and matching. The same word typed with and without marks is a different string. Strip them from the search key as part of Arabic text normalization, or exact matches fail.
  • Tokens. Every mark is an extra character for the tokenizer. Fully vowelled text uses noticeably more tokens than bare text; see the diacritization guide for measured ratios and Arabic tokens per word for the baseline.
  • Speech and evaluation. Transcripts usually come back without marks. Score word error rate after removing diacritics from both texts, or a correct word with an extra fatha counts as an error.
  • When you need them. Text-to-speech, language teaching, Quranic and poetic text, and names that are ambiguous without vowels.

On BroadNet Inference

BroadNet has no diacritization endpoint. The chat models receive tashkeel as ordinary text through POST /v1/chat/completions, so you can ask a model such as Qwen3.8 27B to add or remove marks. The safe pattern is to check the reply in code: remove the diacritics and compare with your input, letter for letter. The LLM diacritization post has the prompt and the check.

For removal, do not use a model at all. A short regex over U+064B to U+0652 is faster, free and exact. The Arabic text normalizer shows the effect in the browser.

Questions

Should I strip diacritics before indexing Arabic text?

For keyword search, usually yes, on the search key only. Tashkeela's authors note that most search engines ignore diacritics. Keep the original text for display and for the LLM.

Why does diacritized Arabic cost more tokens?

Each mark is a separate Unicode character, so a fully vowelled word has more characters for the tokenizer to split. Count your own text with a token counter before and after adding tashkeel.

Can an LLM add diacritics reliably?

It can add them, but it may also change letters. Strip the marks from the reply and compare with the input; reject any reply where a letter changed.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity