The same three Arabic letters can mean “he wrote”, “books” or “it was written”. Diacritics are what tell them apart, and most text leaves them out.
How it works
The Unicode Standard calls these marks tashkil and says they “may be written as combining marks”, then adds: “In normal writing, however, these marks are omitted.” Harakat is the name for the short-vowel subset. Each mark is its own combining character that follows the letter it sits on. The main ones are in the range U+064B to U+0652:
| Code point | Name | Role |
|---|---|---|
| U+064E | FATHA | short a |
| U+064F | DAMMA | short u |
| U+0650 | KASRA | short i |
| U+064B to U+064D | FATHATAN, DAMMATAN, KASRATAN | tanween (-an, -un, -in endings) |
| U+0651 | SHADDA | doubled consonant |
| U+0652 | SUKUN | no vowel |
A worked example with the bare word كتب:
- كَتَبَ (kataba), “he wrote”
- كُتُب (kutub), “books”
- كُتِبَ (kutiba), “it was written”
Without marks, all three are the same string. A reader picks the meaning from the sentence. Software has to do the same, which is the job of a diacritizer. Zerrouki and Balla (2017) built the Tashkeela corpus, about 75.6 million words, mostly Classical Arabic, because fully vowelled text is scarce and diacritizers need it for training. Fadel et al. (2019) report neural diacritization models that match or beat earlier work on a public benchmark, and show that diacritics can help machine translation.
Libraries treat these marks as one set. CAMeL Tools’ AR_DIAC_CHARSET holds U+064B to U+0652 plus U+0670 (superscript alef), and its dediac_ar function removes them.
Why it matters
- Search and matching. The same word typed with and without marks is a different string. Strip them from the search key as part of Arabic text normalization, or exact matches fail.
- Tokens. Every mark is an extra character for the tokenizer. Fully vowelled text uses noticeably more tokens than bare text; see the diacritization guide for measured ratios and Arabic tokens per word for the baseline.
- Speech and evaluation. Transcripts usually come back without marks. Score word error rate after removing diacritics from both texts, or a correct word with an extra fatha counts as an error.
- When you need them. Text-to-speech, language teaching, Quranic and poetic text, and names that are ambiguous without vowels.
On BroadNet Inference
BroadNet has no diacritization endpoint. The chat models receive tashkeel as ordinary text through POST /v1/chat/completions, so you can ask a model such as Qwen3.8 27B to add or remove marks. The safe pattern is to check the reply in code: remove the diacritics and compare with your input, letter for letter. The LLM diacritization post has the prompt and the check.
For removal, do not use a model at all. A short regex over U+064B to U+0652 is faster, free and exact. The Arabic text normalizer shows the effect in the browser.