---
title: "What is Arabic text normalization? Rules for search and RAG"
description: "Arabic text normalization folds spelling variants such as alef forms, tatweel and diacritics into one form so search, RAG and WER scoring match. Rules, trade-offs."
url: https://broadrouter.com/glossary/arabic-text-normalization/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Arabic text normalization

Updated 2026-11-15

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

Also called: Arabic normalization, alef normalization, orthographic normalization, tatweel, kashida, alef variants

Arabic users type the same word several ways. Normalization makes those spellings match.

## How it works

Normalization is a set of character rewrites applied to both stored text and queries. Apache Lucene’s ArabicNormalizer, used in many search engines, lists five rules:

1.  hamza with an alef seat to bare alef;
2.  teh marbuta (ة) to heh (ه);
3.  dotless yeh, alef maksura (ى), to yeh (ي);
4.  removal of the harakat ([diacritics](https://broadrouter.com/glossary/arabic-diacritics/));
5.  removal of tatweel, the “stretching character”.

CAMeL Tools offers the same ideas as separate functions. Its `normalize_alef_ar` maps إ أ ٱ آ (U+0625, U+0623, U+0671, U+0622) to ا, `normalize_alef_maksura_ar` maps ى to ي, and `normalize_teh_marbuta_ar` maps ة to ه. `normalize_unicode` runs NFKC by default, which is [Unicode normalization](https://broadrouter.com/glossary/unicode-normalization/) and should come first.

Tatweel (U+0640, also called kashida) is a joining character the Unicode Standard describes for justification: inserting it between characters stretches a line. It carries no meaning, so كتـــاب and كتاب should match.

A worked example:

| Input | After normalization | Rules used |
| --- | --- | --- |
| إِسْلامٌ | اسلام | diacritics, alef |
| مدرسة | مدرسه | teh marbuta |
| مستشفى | مستشفي | alef maksura |
| كتـــاب | كتاب | tatweel |

## Why it matters

Larkey, Ballesteros and Connell (2002) normalized the TREC-2001 Arabic newswire collection by removing punctuation, diacritics and non-letters and replacing several letter variants with one form. Normalization alone raised monolingual average precision from .194 to .238. Their light8 stemmer on top reached .376.

The same applies to:

-   **[RAG](https://broadrouter.com/glossary/rag/) and keyword search.** A query typed without hamza should find a document written with it. See [Arabic RAG search](https://broadrouter.com/blog/arabic-rag-search/).
-   **[Word error rate](https://broadrouter.com/glossary/word-error-rate/).** Arabic ASR scores depend on whether references and transcripts were normalized the same way. The [Arabic WER explainer](https://broadrouter.com/blog/arabic-asr-wer-explained/) shows the effect.
-   **Deduplication and analytics.** Counting words or users’ terms without normalization splits one word into several.

The cost is merged words. Unifying alef turns the distinct particles إن and أن into the same string, and the ة to ه rule merges كرة (ball) with كره (hatred). Treat [hamza](https://broadrouter.com/glossary/hamza/) and [taa marbuta](https://broadrouter.com/glossary/taa-marbuta/) rules as tunable.

## On BroadNet Inference

Normalization runs in your code, not on the API. BroadNet’s chat and transcription endpoints take your text or audio as given. Keep two copies: the original for the model and the user, and a normalized key for matching.

The free [Arabic text normalizer](https://broadrouter.com/tools/arabic-text-normalizer/) applies these rules in the browser so you can see what each one changes. The [normalization guide](https://broadrouter.com/blog/arabic-text-normalization-search/) has tested Python code and a rule-by-rule discussion.

## Common mistakes

-   Normalizing documents but not queries, or the reverse.
-   Showing or citing the normalized text to users.
-   Mixing libraries whose rules differ, such as two different mappings for ى.

## Questions

### Does normalization help embedding search too?

The classic results were measured on keyword search. For embeddings, test normalized and raw input on your own queries before deciding.

### Should I normalize text before sending it to an LLM?

Usually not. Send the original so the model sees real spelling, hamza and diacritics. Normalize only the copies you match, index or score.

### Do all libraries apply the same rules?

No. CAMeL Tools and Lucene both map alef maksura to yeh and teh marbuta to heh, but other libraries differ. Pick one rule set and use it on both documents and queries.

## See also

-   [Arabic diacritics (tashkeel)](https://broadrouter.com/glossary/arabic-diacritics/) (term)
-   [Hamza](https://broadrouter.com/glossary/hamza/) (term)
-   [Taa marbuta](https://broadrouter.com/glossary/taa-marbuta/) (term)
-   [Unicode normalization](https://broadrouter.com/glossary/unicode-normalization/) (term)
-   [RAG (retrieval-augmented generation)](https://broadrouter.com/glossary/rag/) (term)
-   [Word error rate](https://broadrouter.com/glossary/word-error-rate/) (term)
-   [Arabic text normalization for search and RAG: what to change, what not](https://broadrouter.com/blog/arabic-text-normalization-search/) (article)
-   [Arabic RAG: normalisation, chunking and retrieval that works](https://broadrouter.com/blog/arabic-rag-search/) (article)
-   [Arabic ASR word error rate (WER) explained](https://broadrouter.com/blog/arabic-asr-wer-explained/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-arabic-text-normalization&utm_content=page-cta) ·

**Sources**

-   [CAMeL Tools docs, camel\_tools.utils.normalize (checked 2026-11-15)](https://camel-tools.readthedocs.io/en/latest/api/utils/normalize.html)
-   [CAMeL Tools source, normalize.py (alef, alef maksura, teh marbuta rules) (checked 2026-11-15)](https://raw.githubusercontent.com/CAMeL-Lab/camel_tools/master/camel_tools/utils/normalize.py)
-   [Apache Lucene 9.0 API, ArabicNormalizer (checked 2026-11-15)](https://lucene.apache.org/core/9_0_0/analysis/common/org/apache/lucene/analysis/ar/ArabicNormalizer.html)
-   [Larkey, Ballesteros and Connell, Improving Stemming for Arabic Information Retrieval (SIGIR 2002) (checked 2026-11-15)](https://ciir-publications.cs.umass.edu/pub/web/getpdf.php?id=249)
-   [The Unicode Standard 18.0, Chapter 9 Middle East-I, Arabic (tatweel) (checked 2026-11-15)](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-9/)
