Arabic classification jobs are everywhere: is this review positive, is this WhatsApp message a complaint, which team should get this ticket, is it urgent. Before LLMs, each job needed a labelled dataset and a fine-tuned model. Now you can start with a prompt. That is faster, but it is easy to ship a classifier that looks good on ten examples and fails on Gulf dialect. This post covers the prompt, the output format, the test and the cost.
Which classification jobs suit an LLM?
| Job | Labels, for example | Notes |
|---|---|---|
| Sentiment | positive, negative, neutral, mixed | Neutral and mixed are where people and models disagree most |
| Intent | complaint, question, order, cancel, other | Keep “other”; forcing a label creates errors |
| Ticket routing | billing, delivery, technical, account | Base labels on your real teams |
| Urgency | urgent, normal | Define urgent in one line: “customer cannot use the service now” |
| Dialect or script | MSA, Gulf, Egyptian, Levantine, Arabizi | Useful for routing to the right reply template; see Arabizi |
| Topic tags | Several labels from a fixed list | Ask for a list, cap its length |
An LLM is a good fit when you have few labelled examples, when labels change often, or when you want a reason with each label. A fine-tuned classifier is a good fit when labels are stable, volume is huge and you already have thousands of labelled examples.
What does the research say about LLMs on Arabic?
Two results are worth knowing before you start:
- Fine-tuned Arabic models can win. GPTAraEval tested ChatGPT on 44 Arabic tasks and found it “consistently surpassed by smaller models that have undergone finetuning on Arabic”. The authors also found weaker results on dialects than on Modern Standard Arabic (Khondaker et al.).
- Dialect-aware prompts help. A RANLP 2025 study of DeepSeek-R1, Qwen2.5 and LLaMA-3 on sentiment in MSA, Saudi dialect and Moroccan Darija reports that dialect-aware prompting improved accuracy, especially for the neutral and negative classes (Alharbi et al.).
These are older or different models from the ones you will use, so treat them as directions, not numbers. The lesson holds: measure on your own messages, per dialect.
How do you design the labels?
Start here, not with the model.
- Keep the list short. Four to eight labels per question. Ask separate questions (sentiment, intent, urgency) as separate fields, not one giant list.
- Define each label in one line, with what it is not: “complaint: the customer reports a problem with something they bought. Not a question about price.”
- Add an escape label such as
otherorunclear. - Decide the mixed cases up front. “The delivery was late but the driver was kind” is mixed, not neutral. Write that rule down.
Label definitions can be in English even when the text is Arabic. Keep the label values in English so your code does not depend on Arabic string matching.
What prompt works for Arabic classification?
import json, os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
LABELS = ["complaint", "question", "order", "cancel", "other"]
SYSTEM = """You classify customer messages written in Arabic.
Messages may be Modern Standard Arabic, Gulf or Egyptian dialect, Arabizi, or mixed with English.
Labels:
- complaint: reports a problem with a product, delivery or service.
- question: asks for information; no problem reported.
- order: wants to buy or change an order.
- cancel: wants to cancel an order or subscription.
- other: none of the above.
Return JSON only: {"label": one of the labels, "evidence": the shortest Arabic phrase from the message that decided it}.
Examples:
"الطلب وصل ناقص قطعتين" -> {"label": "complaint", "evidence": "وصل ناقص"}
"بكم التوصيل للرياض؟" -> {"label": "question", "evidence": "بكم التوصيل"}
"عايز ألغي الاشتراك" -> {"label": "cancel", "evidence": "ألغي الاشتراك"}"""
def classify(text):
r = client.chat.completions.create(
model="broadnet/qwen3.8-27b",
temperature=0,
reasoning_effort="low",
max_tokens=1000,
messages=[{"role": "system", "content": SYSTEM},
{"role": "user", "content": text}])
out = json.loads(r.choices[0].message.content)
if out.get("label") not in LABELS:
raise ValueError(f"bad label: {out}")
return out
Why each part is there:
- The dialect line tells the model what to expect, which is the dialect-aware prompting the RANLP study found helpful.
- Examples in more than one dialect. The examples above mix Gulf and Egyptian dialect. Use real (anonymised) messages from your own traffic.
- Evidence as a quote lets a reviewer check a label in a second, and makes guesses easier to spot.
reasoning_effort="low". Qwen3.8 27B, GLM 5.3 and DeepSeek V4.1 Flash reason before answering, and reasoning counts towardmax_tokens. BroadNet’s docs suggest settingreasoning_efforttolowor leaving 1,000 tokens or more. Why answers come back empty otherwise.- Reject unknown labels in code. Never trust the model to stay on the list.
For a stricter output contract, see JSON mode vs json_schema. BroadNet lists structured output for GLM 5.3 and Qwen3.8 27B.
Which BroadNet model should you use?
| Model | Why pick it | Input / output per 1M tokens |
|---|---|---|
| Qwen3.8 27B | Tested by BroadNet in Arabic: Gulf and Egyptian dialects understood; structured output listed | $0.175 / $2.17 |
| BroadNet Gemma 4 VL 26B | Lowest per-token price on BroadNet; runs on the $5 starting credit; structured output not listed, so validate output strictly | $0.10 / $0.22 |
| DeepSeek V4.1 Flash | Also on a flat plan at $60 a month per session, one request at a time | $0.175 / $0.52 |
| GLM 5.3 | Structured output listed; highest price of the four | $0.78 / $3.46 |
Run your test set on Qwen3.8 27B and Gemma 4 VL 26B first. If Gemma’s scores are close enough, it is the cheaper choice for short labels. Pick on your numbers, not on this table.
How do you test the classifier?
- Label 200 real messages by hand, with a native speaker, using the label definitions above. Include every dialect you receive, at least 30 of each.
- Run the model on all 200.
- Compute macro F1, the average of the per-label F1 scores, so rare labels count as much as common ones. Accuracy alone hides a model that never says
cancel. - Read the confusion matrix. If
questionandcomplaintswap often, the definitions overlap: fix the definitions before you change the model. - Split results by dialect. A model can be strong in MSA and weak in Gulf.
# pip install scikit-learn
from sklearn.metrics import classification_report, confusion_matrix
pred = [classify(t)["label"] for t in texts]
print(classification_report(gold, pred, labels=LABELS, digits=3))
print(confusion_matrix(gold, pred, labels=LABELS))
Rerun the same set every time you change the prompt or the model. How to evaluate an LLM before switching covers keeping a fixed test set.
How do you classify at volume?
Classification is input-heavy: a long system prompt, a short message and a one-word answer. Three levers matter.
Group messages per request. A BroadNet pay-as-you-go key allows 30 requests a minute and 4 in flight. One message per request caps you at 30 messages a minute per key, or 1,440 times that a day. Send 20 short messages as a numbered list and ask for a JSON array of 20 results: the same rate limit then covers twenty times more messages. Check that the array length matches and that each item has its number.
Keep the prompt prefix identical. The system prompt and examples are the same on every call. Put them first and the message last, so they can be read from cache at the cached-input price, such as $0.0435 instead of $0.175 per 1M on Qwen3.8 27B. How to structure prompts so the prefix caches.
Keep output short. Output tokens cost more than input. A label and a short quote is enough; drop the evidence field once you trust the model.
Arabic usually takes more tokens per word than English, so count a sample of your messages in the Arabic token counter, then price the month in the LLM API cost calculator.
What mistakes should you avoid?
- Testing on English translations. Translated text is not how your customers write.
- One score for all dialects. Report each dialect separately.
- No escape label. Forcing every message into a real label inflates errors.
- Free-text answers. “It seems negative” is not parseable. Ask for JSON and validate.
- Ignoring sarcasm. “ما شاء الله، ثالث مرة يتأخر الطلب” reads positive word by word and is a complaint. Put an example like it in the prompt.
When is an LLM the wrong classifier?
When labels are stable, you already have thousands of labelled examples and volume is in the millions, a fine-tuned Arabic encoder model is usually cheaper per message and, per GPTAraEval, can be more accurate. For big offline jobs that can wait, a provider with a batch API may also cost less: OpenAI advertises a 50% discount for batch jobs completed within 24 hours. BroadNet has no batch API. A common pattern is to use an LLM to label the first few thousand messages, check them by hand, and train a small model on the result.
Where BroadNet Inference fits
BroadNet gives you an Arabic-tested model (Qwen3.8 27B), a cheap model on the starting credit (Gemma 4 VL 26B) and a flat-rate option (DeepSeek V4.1 Flash) behind one OpenAI-compatible key, with no message content retained after the request. That suits support teams that want to classify Arabic messages without building a training pipeline. It does not suit very large offline jobs where a batch discount or a fine-tuned model would cost less. Start with the prompt above, the 200-message test, and the Gulf, Egyptian and Levantine dialect post for building the test set.
