---
title: "What is Unicode normalization? NFC vs NFKC for LLM text"
description: "Unicode normalization (NFC, NFD, NFKC, NFKD) makes equivalent strings use the same code points. Which form to use, and why NFKC matters for Arabic PDF and OCR text."
url: https://broadrouter.com/glossary/unicode-normalization/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Unicode normalization

Updated 2026-11-15

Unicode normalization converts text to a standard form so that strings that look the same use the same code points. The four forms are NFD, NFC, NFKD and NFKC; the K forms also fold compatibility characters, such as Arabic presentation forms and ligatures, into ordinary letters.

Also called: NFC, NFKC, NFD, NFKD

Two strings can look identical on screen and still fail an equality check. Unicode normalization is the fix.

## How it works

UAX #15 defines four forms:

| Form | Definition |
| --- | --- |
| NFD | Canonical decomposition |
| NFC | Canonical decomposition, then canonical composition |
| NFKD | Compatibility decomposition |
| NFKC | Compatibility decomposition, then canonical composition |

Canonical equivalents should look and behave the same, in MDN’s words. Compatibility equivalents “should be treated alike in some — but not necessarily all — applications”, so NFKC and NFKD can change how text looks. JavaScript’s `normalize()` defaults to NFC. Python offers the same four forms in `unicodedata.normalize(form, s)`, and `unicodedata.is_normalized` to test a string.

Arabic shows both kinds.

**Canonical.** The Unicode Standard says أ (U+0623) decomposes to ا (U+0627) plus U+0654 HAMZA ABOVE. Text from some keyboards or converters arrives in the two-character form. NFC recomposes it to U+0623, so it matches.

**Compatibility.** Unicode encodes Arabic presentation forms, contextual letter shapes and ligatures, in U+FB50 to U+FDFF and U+FE70 to U+FEFE, and says they are for compatibility and “should not be used in general interchange”. They still turn up in text copied from older PDFs and some OCR output. NFKC folds them:

| Input | Code point | After NFKC |
| --- | --- | --- |
| ﻻ | U+FEFB, LAM WITH ALEF ISOLATED FORM | لا (two letters) |
| ﷲ | U+FDF2, ALLAH ISOLATED FORM | الله (four letters) |
| ﺍ | U+FE8D, ALEF ISOLATED FORM | ا |

```python
import unicodedata
unicodedata.normalize("NFKC", "ﻻ") == "لا"   # True
unicodedata.normalize("NFC", "أ") == "أ"    # True
```

CAMeL Tools’ `normalize_unicode` applies NFKC by default, after first mapping two special characters, the Rial sign U+FDFC and the Basmala ligature U+FDFD, to plain Arabic letters.

## Why it matters

-   **Search and RAG.** A document extracted from a PDF with presentation forms will not match a query typed on a normal keyboard. Run NFKC on both before the [Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/) rules, which assume standard letters.
-   **Token counts.** Presentation forms are different code points from the ordinary letters, so a [tokenizer](https://broadrouter.com/glossary/tokenizer/) can split the same visible word into a different number of tokens. In UTF-8, ﻻ is 3 bytes and لا is 4. Count both versions in the [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) if your input comes from PDFs.
-   **Validation.** Normalize before comparing model output with source text, for example when checking that an extracted entity really appears in the document.

## On BroadNet Inference

BroadNet’s API reference does not say whether input is normalized on the server, so normalize in your own code before you send, index or compare text. For Arabic documents, the [vision LLM OCR post](https://broadrouter.com/blog/arabic-ocr-vision-llm/) and the [Arabic invoices guide](https://broadrouter.com/guides/read-arabic-invoices-with-vision/) cover reading pages as images with models such as [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/), which avoids broken PDF text layers. Run NFKC on whatever text comes back before you match it.

## Common mistakes

-   Storing NFKC text for display; keep NFC originals.
-   Normalizing documents but not queries.
-   Assuming normalization removes diacritics, tatweel or bidi marks. It does not.

## Questions

### Should I use NFC or NFKC?

NFC for storing and displaying text, since it does not change appearance. NFKC for search keys and matching, where folding ligatures and presentation forms is what you want. MDN notes the compatibility forms change visual appearance.

### Does Unicode normalization remove Arabic diacritics?

No. Diacritics are separate characters and stay. Removing them is a separate step of Arabic text normalization.

### Does normalization remove invisible bidi marks or zero-width characters?

Not the bidi controls such as U+200F or U+2067; they pass through NFC and NFKC unchanged. Strip them separately if they break matching.

## See also

-   [Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/) (term)
-   [Hamza](https://broadrouter.com/glossary/hamza/) (term)
-   [Tokenizer](https://broadrouter.com/glossary/tokenizer/) (term)
-   [Optical character recognition (OCR)](https://broadrouter.com/glossary/ocr/) (term)
-   [Right-to-left (RTL) text](https://broadrouter.com/glossary/rtl-text/) (term)
-   [Token](https://broadrouter.com/glossary/token/) (term)
-   [Read Arabic invoices and receipts with a vision model](https://broadrouter.com/guides/read-arabic-invoices-with-vision/) (guide)
-   [Arabic text normalization for search and RAG: what to change, what not](https://broadrouter.com/blog/arabic-text-normalization-search/) (article)
-   [Arabic OCR vs vision LLMs: reading Arabic documents and invoices](https://broadrouter.com/blog/arabic-ocr-vision-llm/) (article)
-   [Tokens per word in Arabic vs English: why Arabic costs more](https://broadrouter.com/blog/tokens-per-word-arabic-english/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-unicode-normalization&utm_content=page-cta) ·

**Sources**

-   [Unicode Standard Annex #15, Unicode Normalization Forms (Unicode 18.0) (checked 2026-11-15)](https://www.unicode.org/reports/tr15/)
-   [MDN Web Docs, String.prototype.normalize() (checked 2026-11-15)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize)
-   [Python documentation, unicodedata (normalize, is\_normalized) (checked 2026-11-15)](https://docs.python.org/3/library/unicodedata.html)
-   [The Unicode Standard 18.0, Chapter 9 Middle East-I, Arabic (presentation forms) (checked 2026-11-15)](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-9/)
-   [CAMeL Tools source, normalize.py (normalize\_unicode) (checked 2026-11-15)](https://raw.githubusercontent.com/CAMeL-Lab/camel_tools/master/camel_tools/utils/normalize.py)
