BroadRouter

Arabic text normalizer

Updated

In short: Arabic normalization maps spelling variants to one form so search terms match: remove tashkeel (diacritics) and tatweel, unify أ إ آ to ا, and optionally ى to ي and ة to ه. Apply the same steps to documents and to queries, and keep the original text for display. Paste text to normalize it and copy a matching Python function.

Free, no sign-up, runs in your browser: nothing you type is sent anywhere.

Text and options
Options

Normalized text
Counts before and after normalization
CountBeforeAfterChange

What each step changed

    Python function with these options
    normalize_arabic()

    Character counts are Unicode code points. Word counts split on spaces. The Python function produces the same output as this page on our test strings (Python 3 standard library only).

    Report a wrong price or number (email to info@broadnet.ai, with this page's link and your inputs filled in).

    How it works

    1. Paste Arabic text, or edit the example. It stays in your browser.
    2. Choose a preset: "Search and RAG index" for aggressive matching or "Light clean-up" for text you will still show or send to an LLM. Then turn single options on or off.
    3. Read the normalized text, the before and after character and word counts and what each step changed. Tick "Count tokens" to add real token counts.
    4. Copy the result, or copy the Python function that applies the same options, and use it for both documents and queries.

    Which normalization steps should I use?

    StepWhyTrade-offUsed by
    Remove tashkeelMost text is written without vowels; vocalized and plain spellings should matchLoses distinctions such as عَلِمَ / عُلِمَ; keep originals for displayLucene, Larkey et al., CAMeL Tools dediac_ar, PyArabic strip_tashkeel
    Remove tatweelDecorative stretching, no meaningNone for searchLucene, PyArabic strip_tatweel
    Unify alefHamza on alef is often omitted in typingMerges a few distinct words; rarely a problem for retrievalLucene, Larkey et al., CAMeL Tools normalize_alef_ar
    ى → يLoose and dialect typing mixes them, often at word endMerges على and عليLucene, Larkey et al., CAMeL Tools normalize_alef_maksura_ar
    ة → هOften typed as ه, especially in informal textMerges some feminine nouns with pronoun-suffixed wordsLucene, Larkey et al., CAMeL Tools normalize_teh_marbuta_ar
    Hamza seats ؤ ئTyping variationAggressive; changes many correct spellingsPyArabic normalize_hamza (wider scope)
    DigitsMatch ١٢٣ with 123 in IDs, prices and datesReverse conversion also changes digits in codes and URLsCommon in search pipelines
    Zero-width marksInvisible characters break exact matches and token countsZWNJ is meaningful in Persian and UrduUnicode general category Cf

    Larkey, Ballesteros and Connell normalized the TREC 2001 Arabic collection by removing punctuation, diacritics and non-letters, unifying alef forms and replacing final ى and final ة before stemming. In their monolingual results, normalization alone raised average precision from 0.194 to 0.238, a 23% gain, and light stemming on top raised it further. That is one collection from 2002; measure on your own queries.

    Worked example

    أَهْلاً وَسَهْلاً بِكُمْ فِي الـمُـسْتَشْفَى! رقم الموعد: ١٢٣٤ — الساعة ٩:٣٠ صباحاً، هل تحتاج إلى مساعدة؟‏

    With the "Search and RAG index" preset becomes:

    اهلا وسهلا بكم في المستشفي! رقم الموعد: 1234 — الساعه 9:30 صباحا، هل تحتاج الي مساعده؟

    Diacritics, tatweel and the invisible right-to-left mark are gone, أ became ا, ى became ي, ة became ه and the Arabic-Indic digits became 0–9. Apply the same function to user queries so both sides match.

    Where to normalize in a RAG pipeline

    1. Store the original text with each chunk for display and for the LLM prompt.
    2. Index a normalized copy for keyword search, and test whether embeddings work better on the original or the normalized text.
    3. Normalize each query with the same function before searching.

    More in Arabic RAG: normalisation, chunking and retrieval. BroadNet Inference sells no embeddings; it answers from the retrieved passages with its chat models.

    Questions

    What is Arabic text normalization?

    Mapping characters and spelling variants that readers treat as the same word to one form, so a search for احمد also finds أحمد and أَحْمَد. Typical steps remove diacritics and tatweel and unify alef forms; some pipelines also map ى to ي and ة to ه. Apply the same function to documents and queries.

    Should I remove tashkeel before indexing?

    Usually yes for search and retrieval: most Arabic text is written without diacritics, so a vocalized query or document would otherwise miss. Keep the original text for display and for tasks where vowels change meaning, such as Quranic or poetry text, pronunciation or text to speech.

    Is it safe to change ة to ه and ى to ي?

    It is a trade-off. Both merges help when people type loosely, as many do in dialects and on phones, but they also merge real words: على (on) and علي (Ali), or a feminine noun in ة with a pronoun suffix in ه. Lucene's Arabic normalizer applies both; test on your own queries before turning them on.

    Does normalizing Arabic reduce LLM tokens?

    Removing diacritics and tatweel usually reduces the token count, because those characters are tokens or parts of tokens. Count it: tick "Count tokens" to run a real tokenizer on the text before and after. Do not normalize text you show to users or send for generation if spelling matters to the answer.

    Should I normalize text before sending it to an LLM?

    Light normalization helps: remove zero-width characters, tatweel and odd presentation forms that come from PDFs. Removing diacritics or merging letters is for search indexes and embeddings, not for prompts where the model should see the text as written.

    Is my text uploaded?

    No. Normalization runs in your browser. If you count tokens, the page downloads the public tokenizer file once and counts on your device.

    Go deeper

    Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity