Arabic users type the same word several ways. Normalization makes those spellings match.
How it works
Normalization is a set of character rewrites applied to both stored text and queries. Apache Lucene’s ArabicNormalizer, used in many search engines, lists five rules:
- hamza with an alef seat to bare alef;
- teh marbuta (ة) to heh (ه);
- dotless yeh, alef maksura (ى), to yeh (ي);
- removal of the harakat (diacritics);
- removal of tatweel, the “stretching character”.
CAMeL Tools offers the same ideas as separate functions. Its normalize_alef_ar maps إ أ ٱ آ (U+0625, U+0623, U+0671, U+0622) to ا, normalize_alef_maksura_ar maps ى to ي, and normalize_teh_marbuta_ar maps ة to ه. normalize_unicode runs NFKC by default, which is Unicode normalization and should come first.
Tatweel (U+0640, also called kashida) is a joining character the Unicode Standard describes for justification: inserting it between characters stretches a line. It carries no meaning, so كتـــاب and كتاب should match.
A worked example:
| Input | After normalization | Rules used |
|---|---|---|
| إِسْلامٌ | اسلام | diacritics, alef |
| مدرسة | مدرسه | teh marbuta |
| مستشفى | مستشفي | alef maksura |
| كتـــاب | كتاب | tatweel |
Why it matters
Larkey, Ballesteros and Connell (2002) normalized the TREC-2001 Arabic newswire collection by removing punctuation, diacritics and non-letters and replacing several letter variants with one form. Normalization alone raised monolingual average precision from .194 to .238. Their light8 stemmer on top reached .376.
The same applies to:
- RAG and keyword search. A query typed without hamza should find a document written with it. See Arabic RAG search.
- Word error rate. Arabic ASR scores depend on whether references and transcripts were normalized the same way. The Arabic WER explainer shows the effect.
- Deduplication and analytics. Counting words or users’ terms without normalization splits one word into several.
The cost is merged words. Unifying alef turns the distinct particles إن and أن into the same string, and the ة to ه rule merges كرة (ball) with كره (hatred). Treat hamza and taa marbuta rules as tunable.
On BroadNet Inference
Normalization runs in your code, not on the API. BroadNet’s chat and transcription endpoints take your text or audio as given. Keep two copies: the original for the model and the user, and a normalized key for matching.
The free Arabic text normalizer applies these rules in the browser so you can see what each one changes. The normalization guide has tested Python code and a rule-by-rule discussion.
Common mistakes
- Normalizing documents but not queries, or the reverse.
- Showing or citing the normalized text to users.
- Mixing libraries whose rules differ, such as two different mappings for ى.