Two strings can look identical on screen and still fail an equality check. Unicode normalization is the fix.
How it works
UAX #15 defines four forms:
| Form | Definition |
|---|---|
| NFD | Canonical decomposition |
| NFC | Canonical decomposition, then canonical composition |
| NFKD | Compatibility decomposition |
| NFKC | Compatibility decomposition, then canonical composition |
Canonical equivalents should look and behave the same, in MDN’s words. Compatibility equivalents “should be treated alike in some — but not necessarily all — applications”, so NFKC and NFKD can change how text looks. JavaScript’s normalize() defaults to NFC. Python offers the same four forms in unicodedata.normalize(form, s), and unicodedata.is_normalized to test a string.
Arabic shows both kinds.
Canonical. The Unicode Standard says أ (U+0623) decomposes to ا (U+0627) plus U+0654 HAMZA ABOVE. Text from some keyboards or converters arrives in the two-character form. NFC recomposes it to U+0623, so it matches.
Compatibility. Unicode encodes Arabic presentation forms, contextual letter shapes and ligatures, in U+FB50 to U+FDFF and U+FE70 to U+FEFE, and says they are for compatibility and “should not be used in general interchange”. They still turn up in text copied from older PDFs and some OCR output. NFKC folds them:
| Input | Code point | After NFKC |
|---|---|---|
| ﻻ | U+FEFB, LAM WITH ALEF ISOLATED FORM | لا (two letters) |
| ﷲ | U+FDF2, ALLAH ISOLATED FORM | الله (four letters) |
| ﺍ | U+FE8D, ALEF ISOLATED FORM | ا |
import unicodedata
unicodedata.normalize("NFKC", "ﻻ") == "لا" # True
unicodedata.normalize("NFC", "أ") == "أ" # True
CAMeL Tools’ normalize_unicode applies NFKC by default, after first mapping two special characters, the Rial sign U+FDFC and the Basmala ligature U+FDFD, to plain Arabic letters.
Why it matters
- Search and RAG. A document extracted from a PDF with presentation forms will not match a query typed on a normal keyboard. Run NFKC on both before the Arabic text normalization rules, which assume standard letters.
- Token counts. Presentation forms are different code points from the ordinary letters, so a tokenizer can split the same visible word into a different number of tokens. In UTF-8, ﻻ is 3 bytes and لا is 4. Count both versions in the Arabic token counter if your input comes from PDFs.
- Validation. Normalize before comparing model output with source text, for example when checking that an extracted entity really appears in the document.
On BroadNet Inference
BroadNet’s API reference does not say whether input is normalized on the server, so normalize in your own code before you send, index or compare text. For Arabic documents, the vision LLM OCR post and the Arabic invoices guide cover reading pages as images with models such as Gemma 4 VL 26B, which avoids broken PDF text layers. Run NFKC on whatever text comes back before you match it.
Common mistakes
- Storing NFKC text for display; keep NFC originals.
- Normalizing documents but not queries.
- Assuming normalization removes diacritics, tatweel or bidi marks. It does not.