BroadRouter
Get started

Unicode normalization

Updated

Unicode normalization converts text to a standard form so that strings that look the same use the same code points. The four forms are NFD, NFC, NFKD and NFKC; the K forms also fold compatibility characters, such as Arabic presentation forms and ligatures, into ordinary letters.

Also called: NFC, NFKC, NFD, NFKD

Two strings can look identical on screen and still fail an equality check. Unicode normalization is the fix.

How it works

UAX #15 defines four forms:

Form Definition
NFD Canonical decomposition
NFC Canonical decomposition, then canonical composition
NFKD Compatibility decomposition
NFKC Compatibility decomposition, then canonical composition

Canonical equivalents should look and behave the same, in MDN’s words. Compatibility equivalents “should be treated alike in some — but not necessarily all — applications”, so NFKC and NFKD can change how text looks. JavaScript’s normalize() defaults to NFC. Python offers the same four forms in unicodedata.normalize(form, s), and unicodedata.is_normalized to test a string.

Arabic shows both kinds.

Canonical. The Unicode Standard says أ (U+0623) decomposes to ا (U+0627) plus U+0654 HAMZA ABOVE. Text from some keyboards or converters arrives in the two-character form. NFC recomposes it to U+0623, so it matches.

Compatibility. Unicode encodes Arabic presentation forms, contextual letter shapes and ligatures, in U+FB50 to U+FDFF and U+FE70 to U+FEFE, and says they are for compatibility and “should not be used in general interchange”. They still turn up in text copied from older PDFs and some OCR output. NFKC folds them:

Input Code point After NFKC
ﻻ U+FEFB, LAM WITH ALEF ISOLATED FORM لا (two letters)
ﷲ U+FDF2, ALLAH ISOLATED FORM الله (four letters)
ﺍ U+FE8D, ALEF ISOLATED FORM ا
import unicodedata
unicodedata.normalize("NFKC", "ﻻ") == "لا"   # True
unicodedata.normalize("NFC", "أ") == "أ"    # True

CAMeL Tools’ normalize_unicode applies NFKC by default, after first mapping two special characters, the Rial sign U+FDFC and the Basmala ligature U+FDFD, to plain Arabic letters.

Why it matters

  • Search and RAG. A document extracted from a PDF with presentation forms will not match a query typed on a normal keyboard. Run NFKC on both before the Arabic text normalization rules, which assume standard letters.
  • Token counts. Presentation forms are different code points from the ordinary letters, so a tokenizer can split the same visible word into a different number of tokens. In UTF-8, ﻻ is 3 bytes and لا is 4. Count both versions in the Arabic token counter if your input comes from PDFs.
  • Validation. Normalize before comparing model output with source text, for example when checking that an extracted entity really appears in the document.

On BroadNet Inference

BroadNet’s API reference does not say whether input is normalized on the server, so normalize in your own code before you send, index or compare text. For Arabic documents, the vision LLM OCR post and the Arabic invoices guide cover reading pages as images with models such as Gemma 4 VL 26B, which avoids broken PDF text layers. Run NFKC on whatever text comes back before you match it.

Common mistakes

  • Storing NFKC text for display; keep NFC originals.
  • Normalizing documents but not queries.
  • Assuming normalization removes diacritics, tatweel or bidi marks. It does not.

Questions

Should I use NFC or NFKC?

NFC for storing and displaying text, since it does not change appearance. NFKC for search keys and matching, where folding ligatures and presentation forms is what you want. MDN notes the compatibility forms change visual appearance.

Does Unicode normalization remove Arabic diacritics?

No. Diacritics are separate characters and stay. Removing them is a separate step of Arabic text normalization.

Does normalization remove invisible bidi marks or zero-width characters?

Not the bidi controls such as U+200F or U+2067; they pass through NFC and NFKC unchanged. Strip them separately if they break matching.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity