---
title: "Arabic text normalizer for search and RAG: remove tashkeel"
description: "Normalize Arabic text for search, RAG and embeddings: remove tashkeel and tatweel, unify alef, ya and ta marbuta, convert digits and strip zero-width characters."
url: https://broadrouter.com/tools/arabic-text-normalizer/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Arabic text normalizer

Updated 2026-11-15

**In short:** Arabic normalization maps spelling variants to one form so search terms match: remove tashkeel (diacritics) and tatweel, unify أ إ آ to ا, and optionally ى to ي and ة to ه. Apply the same steps to documents and to queries, and keep the original text for display. Paste text to normalize it and copy a matching Python function.

Free, no sign-up, runs in your browser: nothing you type is sent anywhere.

Text and optionsArabic textأَهْلاً وَسَهْلاً بِكُمْ فِي الـمُـسْتَشْفَى! رقم الموعد: ١٢٣٤ — الساعة ٩:٣٠ صباحاً، هل تحتاج إلى مساعدة؟‏PresetSearch and RAG indexLight clean-up (display, LLM prompts)CustomDigitsKeep as writtenArabic-Indic and Persian → 0–90–9 → Arabic-Indic (٠–٩)PunctuationKeepArabic marks to ASCII: ، ؛ ؟ ٪Remove all punctuation

Options

 Remove tashkeel (diacritics) Fatha, damma, kasra, tanween, sukun, shadda, dagger alef and Quranic marks Keep shadda (ّ)

 Remove tatweel (ـ) U+0640, a stretching character with no meaning

 Unify alef: أ إ آ ٱ to ا Hamza on alef is often left out when typing

 Alef maqsura to ya: ى to ي Merges على (on) with علي (Ali)

 Ta marbuta to ha: ة to ه Merges مدرسة with مدرسه; also some real word pairs

 Hamza seats: ؤ to و, ئ to ي More aggressive; off in most analyzers

 Persian kaf and yeh: ک to ك, ی to ي Text typed on Persian or Urdu keyboards

 Remove zero-width and direction marks ZWSP, ZWJ, LRM, RLM, ALM, BOM, soft hyphen, bidi controls Keep ZWNJ (needed in Persian and Urdu)

 Unicode NFKC (presentation forms) ﻻ to لا and other legacy shapes from PDFs; NFC is always applied

 Tidy whitespace One space between words, at most one blank line

**Normalized text**

Counts before and after normalization
| Count | Before | After | Change |
| --- | --- | --- | --- |

 Count tokens with a real tokenizerDeepSeek V4.1 Flash (2 MB download)GLM 5.3 (20 MB download)Qwen3.8 27B (13 MB download)Gemma 4 VL 26B (32 MB download)GPT (o200k) (1.1 MB download)

## What each step changed

### Python function with these options

**normalize\_arabic()**

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=tools-arabic-text-normalizer&utm_content=tool) ·

Character counts are Unicode code points. Word counts split on spaces. The Python function produces the same output as this page on our test strings (Python 3 standard library only).

[Report a wrong price or number](mailto:info@broadnet.ai?subject=Wrong%20price%20or%20number%20on%20Arabic%20text%20normalizer&body=Page%3A%20https%3A%2F%2Fbroadrouter.com%2Ftools%2Farabic-text-normalizer%2F%0A%0AWhat%20looks%20wrong%20(model%2C%20figure%20shown%2C%20correct%20figure%20and%20a%20source%20link%20if%20you%20have%20one)%3A%0A) (email to info@broadnet.ai, with this page's link and your inputs filled in).

## How it works

1.  Paste Arabic text, or edit the example. It stays in your browser.
2.  Choose a preset: "Search and RAG index" for aggressive matching or "Light clean-up" for text you will still show or send to an LLM. Then turn single options on or off.
3.  Read the normalized text, the before and after character and word counts and what each step changed. Tick "Count tokens" to add real token counts.
4.  Copy the result, or copy the Python function that applies the same options, and use it for both documents and queries.

## Which normalization steps should I use?

| Step | Why | Trade-off | Used by |
| --- | --- | --- | --- |
| Remove tashkeel | Most text is written without vowels; vocalized and plain spellings should match | Loses distinctions such as عَلِمَ / عُلِمَ; keep originals for display | Lucene, Larkey et al., CAMeL Tools `dediac_ar`, PyArabic `strip_tashkeel` |
| Remove tatweel | Decorative stretching, no meaning | None for search | Lucene, PyArabic `strip_tatweel` |
| Unify alef | Hamza on alef is often omitted in typing | Merges a few distinct words; rarely a problem for retrieval | Lucene, Larkey et al., CAMeL Tools `normalize_alef_ar` |
| ى → ي | Loose and dialect typing mixes them, often at word end | Merges على and علي | Lucene, Larkey et al., CAMeL Tools `normalize_alef_maksura_ar` |
| ة → ه | Often typed as ه, especially in informal text | Merges some feminine nouns with pronoun-suffixed words | Lucene, Larkey et al., CAMeL Tools `normalize_teh_marbuta_ar` |
| Hamza seats ؤ ئ | Typing variation | Aggressive; changes many correct spellings | PyArabic `normalize_hamza` (wider scope) |
| Digits | Match ١٢٣ with 123 in IDs, prices and dates | Reverse conversion also changes digits in codes and URLs | Common in search pipelines |
| Zero-width marks | Invisible characters break exact matches and token counts | ZWNJ is meaningful in Persian and Urdu | Unicode general category Cf |

Larkey, Ballesteros and Connell normalized the TREC 2001 Arabic collection by removing punctuation, diacritics and non-letters, unifying alef forms and replacing final ى and final ة before stemming. In their monolingual results, normalization alone raised average precision from 0.194 to 0.238, a 23% gain, and light stemming on top raised it further. That is one collection from 2002; measure on your own queries.

## Worked example

أَهْلاً وَسَهْلاً بِكُمْ فِي الـمُـسْتَشْفَى! رقم الموعد: ١٢٣٤ — الساعة ٩:٣٠ صباحاً، هل تحتاج إلى مساعدة؟‏

With the "Search and RAG index" preset becomes:

اهلا وسهلا بكم في المستشفي! رقم الموعد: 1234 — الساعه 9:30 صباحا، هل تحتاج الي مساعده؟

Diacritics, tatweel and the invisible right-to-left mark are gone, أ became ا, ى became ي, ة became ه and the Arabic-Indic digits became 0–9. Apply the same function to user queries so both sides match.

## Where to normalize in a RAG pipeline

1.  Store the original text with each chunk for display and for the LLM prompt.
2.  Index a normalized copy for keyword search, and test whether embeddings work better on the original or the normalized text.
3.  Normalize each query with the same function before searching.

More in [Arabic RAG: normalisation, chunking and retrieval](https://broadrouter.com/blog/arabic-rag-search/). BroadNet Inference sells no embeddings; it answers from the retrieved passages with its chat models.

## Questions

### What is Arabic text normalization?

Mapping characters and spelling variants that readers treat as the same word to one form, so a search for احمد also finds أحمد and أَحْمَد. Typical steps remove diacritics and tatweel and unify alef forms; some pipelines also map ى to ي and ة to ه. Apply the same function to documents and queries.

### Should I remove tashkeel before indexing?

Usually yes for search and retrieval: most Arabic text is written without diacritics, so a vocalized query or document would otherwise miss. Keep the original text for display and for tasks where vowels change meaning, such as Quranic or poetry text, pronunciation or text to speech.

### Is it safe to change ة to ه and ى to ي?

It is a trade-off. Both merges help when people type loosely, as many do in dialects and on phones, but they also merge real words: على (on) and علي (Ali), or a feminine noun in ة with a pronoun suffix in ه. Lucene's Arabic normalizer applies both; test on your own queries before turning them on.

### Does normalizing Arabic reduce LLM tokens?

Removing diacritics and tatweel usually reduces the token count, because those characters are tokens or parts of tokens. Count it: tick "Count tokens" to run a real tokenizer on the text before and after. Do not normalize text you show to users or send for generation if spelling matters to the answer.

### Should I normalize text before sending it to an LLM?

Light normalization helps: remove zero-width characters, tatweel and odd presentation forms that come from PDFs. Removing diacritics or merging letters is for search indexes and embeddings, not for prompts where the model should see the text as written.

### Is my text uploaded?

No. Normalization runs in your browser. If you count tokens, the page downloads the public tokenizer file once and counts on your device.

## Go deeper

-   [Arabic RAG: normalisation, chunking and retrieval](https://broadrouter.com/blog/arabic-rag-search/)
-   [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/)
-   [Tokens per word in Arabic and English](https://broadrouter.com/blog/tokens-per-word-arabic-english/)
-   [RAG chunking calculator](https://broadrouter.com/tools/rag-chunking-calculator/)

## More free tools

-   [**LLM API cost calculator**Price a workload on every model at once, with cache hits, reasoning and a flat-plan check.](https://broadrouter.com/tools/llm-api-cost-calculator/)
-   [**Speech-to-text cost calculator**Hours, calls or voice notes to a monthly transcription bill, against your current provider.](https://broadrouter.com/tools/speech-to-text-cost-calculator/)
-   [**LLM token counter**Exact token counts with each model's real tokenizer, with a token view and cost.](https://broadrouter.com/tools/token-counter/)
-   [**Arabic token counter**Arabic vs English token counts per tokenizer: the "Arabic tax" in tokens and money.](https://broadrouter.com/tools/arabic-token-counter/)
-   [**Tokens to words converter**Tokens ↔ words ↔ characters ↔ pages, by language and content type.](https://broadrouter.com/tools/tokens-to-words/)
-   [**Context window calculator**Will it fit? Window used per model, room left for output, cost to send.](https://broadrouter.com/tools/context-window-calculator/)
-   [**Flat-rate vs per-token LLM cost calculator**Break-even between an unlimited plan and paying per token, per agent.](https://broadrouter.com/tools/flat-rate-vs-per-token-calculator/)
-   [**Prompt caching savings calculator**Bill with and without caching, by prefix share and hit rate.](https://broadrouter.com/tools/prompt-caching-calculator/)
-   [**Image token calculator**Tokens and cost per image under each model's documented rule. Drop an image; nothing is uploaded.](https://broadrouter.com/tools/image-token-calculator/)
-   [**Self-hosting vs API cost calculator**GPU rental against your current API and BroadNet per-token and flat-plan prices: cost per 1M tokens and break-even volume.](https://broadrouter.com/tools/self-hosting-vs-api-calculator/)
-   [**Embedding cost calculator**Index, update and query costs for a corpus at OpenAI, Google, Voyage, Mistral and Jina list prices.](https://broadrouter.com/tools/embedding-cost-calculator/)
-   [**LLM rate limit and concurrency calculator**Which limit binds first: RPM, TPM or concurrency, with Little's law.](https://broadrouter.com/tools/llm-rate-limit-calculator/)
-   [**JSON schema generator for LLM structured output**Sample JSON to strict schema, Pydantic, Zod and a ready request.](https://broadrouter.com/tools/json-schema-generator/)
-   [**OpenAI-compatible API request builder**One request, copied as curl, Python, Node or the OpenAI SDKs.](https://broadrouter.com/tools/api-request-builder/)
-   [**Audio file splitter for speech-to-text limits**Chunk count, overlap and copy-ready ffmpeg commands to fit a transcription file-size limit.](https://broadrouter.com/tools/audio-splitter-ffmpeg/)
-   [**Agent run cost estimator**Context growth per step, total tokens and cost of one agent run, with a DeepSeek session comparison.](https://broadrouter.com/tools/agent-run-cost-estimator/)
-   [**OpenAI to BroadNet migration helper**Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.](https://broadrouter.com/tools/openai-to-broadnet-migration/)
-   [**RAG chunking calculator**Chunks, embedding tokens, input per query, context check and monthly LLM cost for a RAG pipeline.](https://broadrouter.com/tools/rag-chunking-calculator/)
-   [**max\_tokens calculator for reasoning models**Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.](https://broadrouter.com/tools/reasoning-token-budget/)
-   [**LLM batch job planner**Run time, binding limit, cost per model and the flat sessions needed to finish a big job by a deadline.](https://broadrouter.com/tools/llm-batch-job-planner/)
-   [**LLM JSON output validator**Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.](https://broadrouter.com/tools/llm-json-output-validator/)
-   [**Prompt token diff**Two prompt versions side by side: changed lines, exact token delta and the monthly cost difference.](https://broadrouter.com/tools/prompt-token-diff/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=tools-arabic-text-normalizer&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference machine-readable prices (pricing.json)](https://inference.broadnet.ai/pricing.json?utm_source=broadrouter&utm_medium=referral&utm_campaign=tools-arabic-text-normalizer&utm_content=sources)
-   [Apache Lucene ArabicNormalizer: hamza on alef, teh marbuta, alef maksura, diacritics and tatweel (checked 2026-11-15)](https://lucene.apache.org/core/9_11_0/analysis/common/org/apache/lucene/analysis/ar/ArabicNormalizer.html)
-   [Larkey, Ballesteros and Connell, Improving Stemming for Arabic Information Retrieval, SIGIR 2002 (normalization steps, section 4.1; checked 2026-11-15)](https://ciir-publications.cs.umass.edu/pub/web/getpdf.php?id=249)
-   [CAMeL Tools: camel\_tools.utils.normalize (alef, alef maksura, teh marbuta; checked 2026-11-15)](https://camel-tools.readthedocs.io/en/latest/api/utils/normalize.html)
-   [CAMeL Tools: camel\_tools.utils.dediac (diacritic removal; checked 2026-11-15)](https://camel-tools.readthedocs.io/en/latest/api/utils/dediac.html)
-   [PyArabic: strip\_tashkeel, strip\_tatweel, normalize\_hamza (checked 2026-11-15)](https://pyarabic.readthedocs.io/ar/latest/README.html)
-   [Unicode Arabic code chart, U+0600 to U+06FF](https://www.unicode.org/charts/PDF/U0600.pdf)
