BroadRouter
Get started

Arabic text classification with an LLM: sentiment, intent, triage

By the BroadNet Inference team · 7 min read · Published · Updated

To classify Arabic text with an LLM, define a short label set with one-line definitions, give a few labelled examples in each dialect you see, ask for JSON with one label and a short quote as evidence, and set reasoning low. Then score the model on 200 hand-labelled messages with macro F1 before trusting it. Fine-tuned Arabic models can still beat a general LLM.

Arabic classification jobs are everywhere: is this review positive, is this WhatsApp message a complaint, which team should get this ticket, is it urgent. Before LLMs, each job needed a labelled dataset and a fine-tuned model. Now you can start with a prompt. That is faster, but it is easy to ship a classifier that looks good on ten examples and fails on Gulf dialect. This post covers the prompt, the output format, the test and the cost.

Which classification jobs suit an LLM?

Job Labels, for example Notes
Sentiment positive, negative, neutral, mixed Neutral and mixed are where people and models disagree most
Intent complaint, question, order, cancel, other Keep “other”; forcing a label creates errors
Ticket routing billing, delivery, technical, account Base labels on your real teams
Urgency urgent, normal Define urgent in one line: “customer cannot use the service now”
Dialect or script MSA, Gulf, Egyptian, Levantine, Arabizi Useful for routing to the right reply template; see Arabizi
Topic tags Several labels from a fixed list Ask for a list, cap its length

An LLM is a good fit when you have few labelled examples, when labels change often, or when you want a reason with each label. A fine-tuned classifier is a good fit when labels are stable, volume is huge and you already have thousands of labelled examples.

What does the research say about LLMs on Arabic?

Two results are worth knowing before you start:

  • Fine-tuned Arabic models can win. GPTAraEval tested ChatGPT on 44 Arabic tasks and found it “consistently surpassed by smaller models that have undergone finetuning on Arabic”. The authors also found weaker results on dialects than on Modern Standard Arabic (Khondaker et al.).
  • Dialect-aware prompts help. A RANLP 2025 study of DeepSeek-R1, Qwen2.5 and LLaMA-3 on sentiment in MSA, Saudi dialect and Moroccan Darija reports that dialect-aware prompting improved accuracy, especially for the neutral and negative classes (Alharbi et al.).

These are older or different models from the ones you will use, so treat them as directions, not numbers. The lesson holds: measure on your own messages, per dialect.

How do you design the labels?

Start here, not with the model.

  1. Keep the list short. Four to eight labels per question. Ask separate questions (sentiment, intent, urgency) as separate fields, not one giant list.
  2. Define each label in one line, with what it is not: “complaint: the customer reports a problem with something they bought. Not a question about price.”
  3. Add an escape label such as other or unclear.
  4. Decide the mixed cases up front. “The delivery was late but the driver was kind” is mixed, not neutral. Write that rule down.

Label definitions can be in English even when the text is Arabic. Keep the label values in English so your code does not depend on Arabic string matching.

What prompt works for Arabic classification?

import json, os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
LABELS = ["complaint", "question", "order", "cancel", "other"]

SYSTEM = """You classify customer messages written in Arabic.
Messages may be Modern Standard Arabic, Gulf or Egyptian dialect, Arabizi, or mixed with English.
Labels:
- complaint: reports a problem with a product, delivery or service.
- question: asks for information; no problem reported.
- order: wants to buy or change an order.
- cancel: wants to cancel an order or subscription.
- other: none of the above.
Return JSON only: {"label": one of the labels, "evidence": the shortest Arabic phrase from the message that decided it}.

Examples:
"الطلب وصل ناقص قطعتين" -> {"label": "complaint", "evidence": "وصل ناقص"}
"بكم التوصيل للرياض؟" -> {"label": "question", "evidence": "بكم التوصيل"}
"عايز ألغي الاشتراك" -> {"label": "cancel", "evidence": "ألغي الاشتراك"}"""

def classify(text):
    r = client.chat.completions.create(
        model="broadnet/qwen3.8-27b",
        temperature=0,
        reasoning_effort="low",
        max_tokens=1000,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": text}])
    out = json.loads(r.choices[0].message.content)
    if out.get("label") not in LABELS:
        raise ValueError(f"bad label: {out}")
    return out

Why each part is there:

  • The dialect line tells the model what to expect, which is the dialect-aware prompting the RANLP study found helpful.
  • Examples in more than one dialect. The examples above mix Gulf and Egyptian dialect. Use real (anonymised) messages from your own traffic.
  • Evidence as a quote lets a reviewer check a label in a second, and makes guesses easier to spot.
  • reasoning_effort="low". Qwen3.8 27B, GLM 5.3 and DeepSeek V4.1 Flash reason before answering, and reasoning counts toward max_tokens. BroadNet’s docs suggest setting reasoning_effort to low or leaving 1,000 tokens or more. Why answers come back empty otherwise.
  • Reject unknown labels in code. Never trust the model to stay on the list.

For a stricter output contract, see JSON mode vs json_schema. BroadNet lists structured output for GLM 5.3 and Qwen3.8 27B.

Which BroadNet model should you use?

Model Why pick it Input / output per 1M tokens
Qwen3.8 27B Tested by BroadNet in Arabic: Gulf and Egyptian dialects understood; structured output listed $0.175 / $2.17
BroadNet Gemma 4 VL 26B Lowest per-token price on BroadNet; runs on the $5 starting credit; structured output not listed, so validate output strictly $0.10 / $0.22
DeepSeek V4.1 Flash Also on a flat plan at $60 a month per session, one request at a time $0.175 / $0.52
GLM 5.3 Structured output listed; highest price of the four $0.78 / $3.46

Run your test set on Qwen3.8 27B and Gemma 4 VL 26B first. If Gemma’s scores are close enough, it is the cheaper choice for short labels. Pick on your numbers, not on this table.

How do you test the classifier?

  1. Label 200 real messages by hand, with a native speaker, using the label definitions above. Include every dialect you receive, at least 30 of each.
  2. Run the model on all 200.
  3. Compute macro F1, the average of the per-label F1 scores, so rare labels count as much as common ones. Accuracy alone hides a model that never says cancel.
  4. Read the confusion matrix. If question and complaint swap often, the definitions overlap: fix the definitions before you change the model.
  5. Split results by dialect. A model can be strong in MSA and weak in Gulf.
# pip install scikit-learn
from sklearn.metrics import classification_report, confusion_matrix
pred = [classify(t)["label"] for t in texts]
print(classification_report(gold, pred, labels=LABELS, digits=3))
print(confusion_matrix(gold, pred, labels=LABELS))

Rerun the same set every time you change the prompt or the model. How to evaluate an LLM before switching covers keeping a fixed test set.

How do you classify at volume?

Classification is input-heavy: a long system prompt, a short message and a one-word answer. Three levers matter.

Group messages per request. A BroadNet pay-as-you-go key allows 30 requests a minute and 4 in flight. One message per request caps you at 30 messages a minute per key, or 1,440 times that a day. Send 20 short messages as a numbered list and ask for a JSON array of 20 results: the same rate limit then covers twenty times more messages. Check that the array length matches and that each item has its number.

Keep the prompt prefix identical. The system prompt and examples are the same on every call. Put them first and the message last, so they can be read from cache at the cached-input price, such as $0.0435 instead of $0.175 per 1M on Qwen3.8 27B. How to structure prompts so the prefix caches.

Keep output short. Output tokens cost more than input. A label and a short quote is enough; drop the evidence field once you trust the model.

Arabic usually takes more tokens per word than English, so count a sample of your messages in the Arabic token counter, then price the month in the LLM API cost calculator.

What mistakes should you avoid?

  • Testing on English translations. Translated text is not how your customers write.
  • One score for all dialects. Report each dialect separately.
  • No escape label. Forcing every message into a real label inflates errors.
  • Free-text answers. “It seems negative” is not parseable. Ask for JSON and validate.
  • Ignoring sarcasm. “ما شاء الله، ثالث مرة يتأخر الطلب” reads positive word by word and is a complaint. Put an example like it in the prompt.

When is an LLM the wrong classifier?

When labels are stable, you already have thousands of labelled examples and volume is in the millions, a fine-tuned Arabic encoder model is usually cheaper per message and, per GPTAraEval, can be more accurate. For big offline jobs that can wait, a provider with a batch API may also cost less: OpenAI advertises a 50% discount for batch jobs completed within 24 hours. BroadNet has no batch API. A common pattern is to use an LLM to label the first few thousand messages, check them by hand, and train a small model on the result.

Where BroadNet Inference fits

BroadNet gives you an Arabic-tested model (Qwen3.8 27B), a cheap model on the starting credit (Gemma 4 VL 26B) and a flat-rate option (DeepSeek V4.1 Flash) behind one OpenAI-compatible key, with no message content retained after the request. That suits support teams that want to classify Arabic messages without building a training pipeline. It does not suit very large offline jobs where a batch discount or a fine-tuned model would cost less. Start with the prompt above, the 200-message test, and the Gulf, Egyptian and Levantine dialect post for building the test set.

Questions

Can an LLM do Arabic sentiment analysis?

Yes, without training, by prompting it with labels and examples. Quality varies by dialect and by class: in published tests, models trail fine-tuned Arabic models on many tasks, and dialects score lower than Modern Standard Arabic. Measure it on your own data.

Is an LLM better than a fine-tuned Arabic BERT model for classification?

Not always. GPTAraEval found ChatGPT "consistently surpassed by smaller models that have undergone finetuning on Arabic" across 44 tasks. An LLM wins when you have few labelled examples, labels that change often, or need a reason with each label.

How do I classify Gulf and Egyptian Arabic messages?

Give the model labelled examples in each dialect, tell it which dialects to expect, and score each dialect separately. A model that does well on Egyptian reviews can still miss Gulf sarcasm.

How do I get a consistent label every time?

Use a fixed label list, ask for JSON with only those values, set temperature to 0, and reject any reply that is not on the list. Send the same examples in the same order so results are repeatable.

How many messages can I classify per day on BroadNet?

A pay-as-you-go key allows 30 requests a minute and 4 at once. At one message per request that is at most 30 messages a minute per key, or 1,440 times that a day. Putting 20 short messages in each request raises that twentyfold, within the context limit.

Does BroadNet have a batch API for bulk classification?

No. BroadNet documents chat completions and transcription only. For large offline jobs, group messages per request, or compare a provider with a batch API; OpenAI's Batch API advertises a 50% discount with a 24-hour window.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity