Arabic proofreading looks like a one-line prompt: “correct this text”. In practice the model also rephrases, adds diacritics, swaps a dialect word for a formal one and explains its changes, and you can’t tell which edits were needed. This post builds a corrector on an OpenAI-compatible LLM that makes the fewest edits, checks the reply in code, labels each edit by error type and scores the whole thing the way the research benchmarks do.
How good are LLMs at Arabic grammar correction?
The standard benchmarks are the QALB shared tasks. QALB-2014 used Modern Standard Arabic comments from Al Jazeera’s website: over one million words of training text with about 306,000 corrections, and test sets of about 50,000 words. QALB-2015 added a second track of essays by learners of Arabic as a foreign language (L2). Both scored systems on edits with the M2 scorer. The best scores, F1 (higher is better):
| Test set | Best system F1 | Text |
|---|---|---|
| QALB-2014 test | 67.91 | Native speakers’ news comments |
| QALB-2015 native test | 72.87 | Native speakers’ news comments |
| QALB-2015 L2 test | 41.20 | Learners’ essays |
The L2 row is the useful warning. Learner text has more lexical choice errors and missing words, and the best system scored far lower. If your users are learners, expect your numbers to look more like that row.
Kwon, Bhatia, Nagoudi and Abdul-Mageed (ArabicNLP 2023) then ran ChatGPT and GPT-4 on the QALB-2014 test set. On the same scorer:
| System | F1 |
|---|---|
| ChatGPT, 1 example | 53.59 |
| ChatGPT, 5 examples | 63.98 |
| GPT-4, 5 examples | 65.49 |
| AraT5v2, fine-tuned | 67.70 |
Two details matter for your prompt. More examples helped a lot. And zero-shot prompts and Arabic-language prompts gave replies full of explanations that the authors could not score automatically. A corrector needs output a program can diff.
Why not just normalise the text?
The Arabic text normalizer on this site removes tashkeel and tatweel, maps أ إ آ to ا and, optionally, ى to ي and ة to ه. That is the right step for search, because a user who types احمد should find أحمد. It is the wrong step before correction, because it erases the distinction the corrector has to restore. After normalisation, انا and أنا are the same string, so the hamza error disappears and so does the fix.
Keep three copies when you need both: the original text, the corrected text you show users, and a normalised copy of the corrected text for your search index. Arabic text normalization for search covers the indexing side. One normalisation step does belong before correction: Unicode NFC, which joins a decomposed alif plus combining hamza into one character, so the diff doesn’t report a change that isn’t there.
What should the prompt say?
Ask for minimal edits, name the error types you care about, forbid the changes you don’t want and ask for the corrected text alone. Writing the instructions in English or Arabic is your choice; Arabic prompt engineering covers the trade-off. Kwon et al. got usable output from English instructions with examples. Adding three to five corrected sentence pairs from your own data (few-shot) is the cheapest improvement to try first.
Set temperature to 0. You want the same correction every time. If the text is a dialect such as Egyptian or Gulf Arabic, say whether to keep the dialect. The prompt below assumes Modern Standard Arabic (MSA) and tells the model not to change style, which also covers dialect.
How do you check the model’s reply?
The code makes one request, measures how much the reply differs from the input, extracts word-level edits and labels each one.
import difflib, os, re, unicodedata
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
SYSTEM = """You are an Arabic proofreader. Correct spelling, grammar and punctuation
errors in the user's Modern Standard Arabic text with the fewest possible edits.
Rules:
- Fix hamza, taa marbuta (ة/ه), alef maqsura (ى/ي), agreement, missing or extra
letters, wrongly split or merged words, and punctuation.
- Do not rephrase, reorder, shorten, translate or change the style or dialect.
- Do not add or remove diacritics (tashkeel).
- If the text is already correct, return it unchanged.
Reply with the corrected text only: no explanation, no quotes, no list of changes."""
def tokens(s: str) -> list[str]:
s = unicodedata.normalize("NFC", s) # composed hamza forms, so أ is one character
return re.findall(r"\w+|[^\w\s]", s) # Arabic letters are \w; ، ؟ ؛ become tokens
def edits(src: str, hyp: str) -> list[tuple]:
"""Word-level edits (start, end, replacement, original): the idea behind an M2 file's A lines."""
a, b = tokens(src), tokens(hyp)
out = []
for op, i1, i2, j1, j2 in difflib.SequenceMatcher(None, a, b, autojunk=False).get_opcodes():
i, j = i1, j1
while op != "equal" and (i < i2 or j < j2): # split the block into one-word edits
if i + 1 < i2 and j < j2 and a[i] + a[i + 1] == b[j]: # و قابلت -> وقابلت
out.append((i, i + 2, b[j], f"{a[i]} {a[i + 1]}")); i += 2; j += 1
elif i < i2 and j + 1 < j2 and a[i] == b[j] + b[j + 1]: # عبدالله -> عبد الله
out.append((i, i + 1, f"{b[j]} {b[j + 1]}", a[i])); i += 1; j += 2
elif i < i2 and j < j2:
out.append((i, i + 1, b[j], a[i])); i += 1; j += 1
elif i < i2: # deleted word
out.append((i, i + 1, "", a[i])); i += 1
else: # inserted word
out.append((i, i, b[j], "")); j += 1
return out
HAMZA = str.maketrans("أإآٱؤئ", "ااااوي")
RULES = [("hamza", lambda w: w.translate(HAMZA).replace("ء", "")),
("taa marbuta", lambda w: w.replace("ة", "ه")),
("alef maqsura", lambda w: w.replace("ى", "ي"))]
def error_class(orig: str, corr: str) -> str:
if all(not c.isalnum() for c in (orig + corr).replace(" ", "")):
return "punctuation"
if orig != corr and orig.replace(" ", "") == corr.replace(" ", ""):
return "merge" if " " in orig else "split"
for name, f in RULES:
if orig != corr and f(orig) == f(corr):
return name
both = lambda w: RULES[2][1](RULES[1][1](RULES[0][1](w)))
if both(orig) == both(corr):
return "mixed orthography"
return "other"
def correct(text: str, model="broadnet/qwen3.8-27b", min_similarity=0.8) -> dict:
r = client.chat.completions.create(
model=model, temperature=0, max_tokens=2000,
messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}])
out = r.choices[0].message.content.strip()
sim = difflib.SequenceMatcher(None, text, out, autojunk=False).ratio()
if sim < min_similarity: # a rewrite, not a correction: keep the original
return {"text": text, "edits": [], "rejected": out, "similarity": round(sim, 2)}
return {"text": out, "similarity": round(sim, 2),
"edits": [{"from": o, "to": c, "class": error_class(o, c)} for _, _, c, o in edits(text, out)]}
We ran it against a local mock server with canned replies (no live API call, since the mock returns whatever we wrote). For ذهبت الى المدرسه و قابلت المعلمه في الصباح, a reply of ذهبت إلى المدرسة وقابلت المعلمة في الصباح gave four edits: الى to إلى (hamza), المدرسه to المدرسة (taa marbuta), و قابلت to وقابلت (merge) and المعلمه to المعلمة (taa marbuta). For مشي الي it listed مشي to مشى (alef maqsura) and الي to إلى (mixed orthography, since it fixes hamza and alef maqsura at once). A canned reply that turned الطقس جميل اليوم into a sentence twice as long scored 0.52 similarity, so the code kept the original and logged the reply.
Notes on the code:
- The similarity guard is a blunt tool. A heavily misspelled sentence can legitimately fall near 0.8. Tune the threshold on your own data, and send rejected replies to a person, not to the bin.
max_tokens=2000leaves room for reasoning. BroadNet’s docs say DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering and that reasoning counts towardmax_tokens; you can also setreasoning_effortto low. See empty responses from reasoning models.- The class labels are rules, not a parser. “Other” includes missing letters (مسؤل to مسؤول), agreement and word choice. For a full taxonomy, run ARETA (below) on the same pairs.
Which error classes should you track?
The QALB organisers name hamzated alifs and alef maqsura/ya confusion as the most common Arabic spelling errors. ARETA, CAMeL Lab’s error-type annotator (Belkebir and Habash, CoNLL 2021), uses a tagset based on the Arabic Learner Corpus. Its orthographic tags include OH (hamza), OT (ha and taa marbuta) and OA (alif and ya), next to morphological, syntactic, semantic, punctuation, merge (MG) and split (SP) tags. Its paper reports 85.8% micro-average F1 against hand-labelled learner text.
| Class | Wrong | Right | What goes wrong |
|---|---|---|---|
| Hamza | انا، الى، مسأله | أنا، إلى، مسألة | Hamza missing, on the wrong seat or on the wrong side of alif |
| Taa marbuta | المدرسه | المدرسة | Final ة written as ه, or the reverse |
| Alef maqsura | مشي، علي (preposition) | مشى، على | Final ى written as ي; علي is also a name, so context decides |
| Merge or split | و قابلت، عبدالله | وقابلت، عبد الله | Spaces around و and ال, compound names |
| Punctuation | missing ، or ؟ | added | QALB annotators agreed least on punctuation |
Track recall per class. A model that fixes taa marbuta but misses hamza after a preposition needs examples of that pattern, not a different model.
How do you score a corrector?
Grammatical error correction is scored on edits. Compare the edits that turn the source into a hand-corrected gold sentence with the edits that turn it into the model’s output. Precision is the share of model edits that match gold. Recall is the share of gold edits the model made. QALB reported F1; later work, including Kwon et al., also reports F0.5, which weights precision twice as much as recall, because a wrong “correction” annoys users more than a missed one.
from collections import Counter
def score(sources, golds, hyps, beta=0.5):
"""Exact-match edit scoring: an edit counts only if span and replacement both match gold."""
tp = fp = fn = 0
by_class = Counter()
for s, g, h in zip(sources, golds, hyps):
G = {e[:3] for e in edits(s, g)}
H = {e[:3] for e in edits(s, h)}
tp += len(G & H); fp += len(H - G); fn += len(G - H)
for e in edits(s, g):
by_class[(error_class(e[3], e[2]), e[:3] in H)] += 1
p = tp / (tp + fp) if tp + fp else 1.0
r = tp / (tp + fn) if tp + fn else 1.0
f = (1 + beta**2) * p * r / (beta**2 * p + r) if p + r else 0.0
recall = {c: f"{by_class[(c, True)]}/{by_class[(c, True)] + by_class[(c, False)]}"
for c in sorted({c for c, _ in by_class})}
return {"P": round(p, 3), "R": round(r, 3), f"F{beta}": round(f, 3), "recall by class": recall}
On a two-sentence toy set with seven gold edits, where the output missed the merge and one taa marbuta and added a full stop, it printed P 0.833, R 0.714 and F0.5 0.806, with recall per class: hamza 2/2, taa marbuta 2/3, merge 0/1.
This is the M2 idea, simplified. The official M2 scorer searches for the alignment of system edits that best matches gold, so a correct fix split into edits differently from gold isn’t penalised. Our exact match is stricter. Use the official scorer when you compare with published numbers (its developer version still needs Python 2). ERRANT does the same edit extraction for English only.
Build the gold set from real user text, 100 to 200 sentences, corrected by one careful annotator with the same minimal-edit rules you gave the model. Run each candidate model and prompt over it, and compare F0.5 and per-class recall. How to evaluate an LLM before switching covers running that comparison fairly.
What about Tibyan and synthetic data?
Labelled Arabic correction data is scarce. Alrehili and Alhothali point out that QALB-14 and QALB-15 together hold about 20,500 parallel examples. Their Tibyan corpus (arXiv 2024, later in PeerJ Computer Science) used ChatGPT to generate error and correct sentence pairs from books on common Arabic errors, reviewed by linguists. It holds about 600,000 tokens, with seven error types: orthography, morphology, syntax, semantics, punctuation, merge and split. The authors say the corpus is public; we did not find a download link to check its licence. It is useful for few-shot examples and stress tests, but it is synthetic, so don’t use it as your acceptance test.
If you need a non-LLM baseline, CAMeL Lab’s arabic-gec repository publishes the code and models from Alhafni et al. (EMNLP 2023). For spelling only, the ar-corrector package suggests dictionary words within one or two edits; it doesn’t handle grammar, and its last release was in 2021.
Which BroadNet model should you use?
Prices are per 1M tokens from BroadNet’s pricing page. Correction output is about as long as the input, so the output price matters.
| Model | Why | Input / output |
|---|---|---|
| Qwen3.8 27B | Reasoning documented; BroadNet reports Arabic testing including Gulf and Egyptian dialect understanding | $0.175 / $2.17 |
| GLM 5.3 | Reasoning documented; long context for whole documents | $0.78 / $3.46 |
| DeepSeek V4.1 Flash | Lowest output price of the reasoning models here; also on flat session plans | $0.175 / $0.52 |
| Gemma 4 VL 26B | Runs on the $5 starting credit, so you can build the pipeline before topping up | $0.10 / $0.22 |
Run the scorer with each model on your gold set and choose on F0.5 and price. Arabic usually takes more tokens per word than English; paste a sample into the Arabic token counter and estimate the bill with the LLM API cost calculator. To proofread a large archive, BroadNet has no Batch API; batch processing without a Batch API shows how to run it with bounded concurrency, or on a DeepSeek session for $15 a week (one request at a time per session).
Where BroadNet Inference fits
BroadRouter is BroadNet’s own site. BroadNet Inference gives OpenAI-compatible access to the models above, so the code runs with the OpenAI Python SDK after changing the base URL. It has no proofreading endpoint and publishes no Arabic correction benchmark of its own, so measure on your text. Its privacy statement says inference content is not retained, which matters when you proofread customer messages. If you need a self-hosted corrector for very high volume, a fine-tuned model such as those from CAMeL Lab may be cheaper and more consistent; the published scores above are the reason to test both.
