BroadRouter
Get started

Arabic text normalization

Updated

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

Also called: Arabic normalization, alef normalization, orthographic normalization, tatweel, kashida, alef variants

Arabic users type the same word several ways. Normalization makes those spellings match.

How it works

Normalization is a set of character rewrites applied to both stored text and queries. Apache Lucene’s ArabicNormalizer, used in many search engines, lists five rules:

  1. hamza with an alef seat to bare alef;
  2. teh marbuta (ة) to heh (ه);
  3. dotless yeh, alef maksura (ى), to yeh (ي);
  4. removal of the harakat (diacritics);
  5. removal of tatweel, the “stretching character”.

CAMeL Tools offers the same ideas as separate functions. Its normalize_alef_ar maps إ أ ٱ آ (U+0625, U+0623, U+0671, U+0622) to ا, normalize_alef_maksura_ar maps ى to ي, and normalize_teh_marbuta_ar maps ة to ه. normalize_unicode runs NFKC by default, which is Unicode normalization and should come first.

Tatweel (U+0640, also called kashida) is a joining character the Unicode Standard describes for justification: inserting it between characters stretches a line. It carries no meaning, so كتـــاب and كتاب should match.

A worked example:

Input After normalization Rules used
إِسْلامٌ اسلام diacritics, alef
مدرسة مدرسه teh marbuta
مستشفى مستشفي alef maksura
كتـــاب كتاب tatweel

Why it matters

Larkey, Ballesteros and Connell (2002) normalized the TREC-2001 Arabic newswire collection by removing punctuation, diacritics and non-letters and replacing several letter variants with one form. Normalization alone raised monolingual average precision from .194 to .238. Their light8 stemmer on top reached .376.

The same applies to:

  • RAG and keyword search. A query typed without hamza should find a document written with it. See Arabic RAG search.
  • Word error rate. Arabic ASR scores depend on whether references and transcripts were normalized the same way. The Arabic WER explainer shows the effect.
  • Deduplication and analytics. Counting words or users’ terms without normalization splits one word into several.

The cost is merged words. Unifying alef turns the distinct particles إن and أن into the same string, and the ة to ه rule merges كرة (ball) with كره (hatred). Treat hamza and taa marbuta rules as tunable.

On BroadNet Inference

Normalization runs in your code, not on the API. BroadNet’s chat and transcription endpoints take your text or audio as given. Keep two copies: the original for the model and the user, and a normalized key for matching.

The free Arabic text normalizer applies these rules in the browser so you can see what each one changes. The normalization guide has tested Python code and a rule-by-rule discussion.

Common mistakes

  • Normalizing documents but not queries, or the reverse.
  • Showing or citing the normalized text to users.
  • Mixing libraries whose rules differ, such as two different mappings for ى.

Questions

Does normalization help embedding search too?

The classic results were measured on keyword search. For embeddings, test normalized and raw input on your own queries before deciding.

Should I normalize text before sending it to an LLM?

Usually not. Send the original so the model sees real spelling, hamza and diacritics. Normalize only the copies you match, index or score.

Do all libraries apply the same rules?

No. CAMeL Tools and Lucene both map alef maksura to yeh and teh marbuta to heh, but other libraries differ. Pick one rule set and use it on both documents and queries.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity