---
title: "Arabic sentiment and ticket classification with an LLM"
description: "Classify Arabic reviews, messages and support tickets with an LLM API: label design, a dialect-aware prompt, JSON output, macro F1 testing and cost at volume."
url: https://broadrouter.com/blog/arabic-text-classification-llm/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Arabic text classification with an LLM: sentiment, intent, triage

By the BroadNet Inference team · 7 min read · Published 2026-11-15 · Updated 2026-11-16

To classify Arabic text with an LLM, define a short label set with one-line definitions, give a few labelled examples in each dialect you see, ask for JSON with one label and a short quote as evidence, and set reasoning low. Then score the model on 200 hand-labelled messages with macro F1 before trusting it. Fine-tuned Arabic models can still beat a general LLM.

## Key takeaways

-   Write the labels and their definitions first. Most classification errors come from vague labels, not from the model.
-   Include examples in each dialect you receive. A 2025 study found dialect-aware prompting improved Arabic sentiment accuracy.
-   Ask for JSON with one label and a quote from the text, and keep the output short: output tokens cost more than input.
-   Score on your own labelled set with macro F1 and a confusion matrix. Neutral and mixed messages are where models disagree with people.
-   Put many short messages in one request to stay inside rate limits: a BroadNet pay-as-you-go key allows 30 requests a minute.

**On this page**

1.  Which classification jobs suit an LLM?
2.  What does the research say about LLMs on Arabic?
3.  How do you design the labels?
4.  What prompt works for Arabic classification?
5.  Which BroadNet model should you use?
6.  How do you test the classifier?
7.  How do you classify at volume?
8.  What mistakes should you avoid?
9.  When is an LLM the wrong classifier?
10.  Where BroadNet Inference fits

Arabic classification jobs are everywhere: is this review positive, is this WhatsApp message a complaint, which team should get this ticket, is it urgent. Before LLMs, each job needed a labelled dataset and a fine-tuned model. Now you can start with a prompt. That is faster, but it is easy to ship a classifier that looks good on ten examples and fails on Gulf dialect. This post covers the prompt, the output format, the test and the cost.

## Which classification jobs suit an LLM?

| Job | Labels, for example | Notes |
| --- | --- | --- |
| Sentiment | positive, negative, neutral, mixed | Neutral and mixed are where people and models disagree most |
| Intent | complaint, question, order, cancel, other | Keep “other”; forcing a label creates errors |
| Ticket routing | billing, delivery, technical, account | Base labels on your real teams |
| Urgency | urgent, normal | Define urgent in one line: “customer cannot use the service now” |
| Dialect or script | MSA, Gulf, Egyptian, Levantine, Arabizi | Useful for routing to the right reply template; see [Arabizi](https://broadrouter.com/blog/arabizi-llm-handling/) |
| Topic tags | Several labels from a fixed list | Ask for a list, cap its length |

An LLM is a good fit when you have few labelled examples, when labels change often, or when you want a reason with each label. A fine-tuned classifier is a good fit when labels are stable, volume is huge and you already have thousands of labelled examples.

## What does the research say about LLMs on Arabic?

Two results are worth knowing before you start:

-   **Fine-tuned Arabic models can win.** GPTAraEval tested ChatGPT on 44 Arabic tasks and found it “consistently surpassed by smaller models that have undergone finetuning on Arabic”. The authors also found weaker results on dialects than on Modern Standard Arabic ([Khondaker et al.](https://arxiv.org/abs/2305.14976)).
-   **Dialect-aware prompts help.** A RANLP 2025 study of DeepSeek-R1, Qwen2.5 and LLaMA-3 on sentiment in MSA, Saudi dialect and Moroccan Darija reports that dialect-aware prompting improved accuracy, especially for the neutral and negative classes ([Alharbi et al.](https://aclanthology.org/2025.ranlp-1.8/)).

These are older or different models from the ones you will use, so treat them as directions, not numbers. The lesson holds: measure on your own messages, per dialect.

## How do you design the labels?

Start here, not with the model.

1.  **Keep the list short.** Four to eight labels per question. Ask separate questions (sentiment, intent, urgency) as separate fields, not one giant list.
2.  **Define each label in one line,** with what it is not: “complaint: the customer reports a problem with something they bought. Not a question about price.”
3.  **Add an escape label** such as `other` or `unclear`.
4.  **Decide the mixed cases up front.** “The delivery was late but the driver was kind” is mixed, not neutral. Write that rule down.

Label definitions can be in English even when the text is Arabic. Keep the label values in English so your code does not depend on Arabic string matching.

## What prompt works for Arabic classification?

```python
import json, os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
LABELS = ["complaint", "question", "order", "cancel", "other"]

SYSTEM = """You classify customer messages written in Arabic.
Messages may be Modern Standard Arabic, Gulf or Egyptian dialect, Arabizi, or mixed with English.
Labels:
- complaint: reports a problem with a product, delivery or service.
- question: asks for information; no problem reported.
- order: wants to buy or change an order.
- cancel: wants to cancel an order or subscription.
- other: none of the above.
Return JSON only: {"label": one of the labels, "evidence": the shortest Arabic phrase from the message that decided it}.

Examples:
"الطلب وصل ناقص قطعتين" -> {"label": "complaint", "evidence": "وصل ناقص"}
"بكم التوصيل للرياض؟" -> {"label": "question", "evidence": "بكم التوصيل"}
"عايز ألغي الاشتراك" -> {"label": "cancel", "evidence": "ألغي الاشتراك"}"""

def classify(text):
    r = client.chat.completions.create(
        model="broadnet/qwen3.8-27b",
        temperature=0,
        reasoning_effort="low",
        max_tokens=1000,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": text}])
    out = json.loads(r.choices[0].message.content)
    if out.get("label") not in LABELS:
        raise ValueError(f"bad label: {out}")
    return out
```

Why each part is there:

-   **The dialect line** tells the model what to expect, which is the dialect-aware prompting the RANLP study found helpful.
-   **Examples in more than one dialect.** The examples above mix Gulf and Egyptian dialect. Use real (anonymised) messages from your own traffic.
-   **Evidence as a quote** lets a reviewer check a label in a second, and makes guesses easier to spot.
-   **`reasoning_effort="low"`.** Qwen3.8 27B, GLM 5.3 and DeepSeek V4.1 Flash reason before answering, and reasoning counts toward `max_tokens`. BroadNet’s docs suggest setting `reasoning_effort` to `low` or leaving 1,000 tokens or more. [Why answers come back empty otherwise](https://broadrouter.com/guides/reasoning-tokens-empty-response/).
-   **Reject unknown labels in code.** Never trust the model to stay on the list.

For a stricter output contract, see [JSON mode vs json\_schema](https://broadrouter.com/blog/structured-output-json-schema/). BroadNet lists structured output for GLM 5.3 and Qwen3.8 27B.

## Which BroadNet model should you use?

| Model | Why pick it | Input / output per 1M tokens |
| --- | --- | --- |
| [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) | Tested by BroadNet in Arabic: Gulf and Egyptian dialects understood; structured output listed | $0.175 / $2.17 |
| [BroadNet Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) | Lowest per-token price on BroadNet; runs on the $5 starting credit; structured output not listed, so validate output strictly | $0.10 / $0.22 |
| [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/) | Also on a flat plan at $60 a month per session, one request at a time | $0.175 / $0.52 |
| [GLM 5.3](https://broadrouter.com/models/glm-5.3/) | Structured output listed; highest price of the four | $0.78 / $3.46 |

Run your test set on Qwen3.8 27B and Gemma 4 VL 26B first. If Gemma’s scores are close enough, it is the cheaper choice for short labels. Pick on your numbers, not on this table.

## How do you test the classifier?

1.  **Label 200 real messages by hand,** with a native speaker, using the label definitions above. Include every dialect you receive, at least 30 of each.
2.  **Run the model on all 200.**
3.  **Compute macro F1,** the average of the per-label F1 scores, so rare labels count as much as common ones. Accuracy alone hides a model that never says `cancel`.
4.  **Read the confusion matrix.** If `question` and `complaint` swap often, the definitions overlap: fix the definitions before you change the model.
5.  **Split results by dialect.** A model can be strong in MSA and weak in Gulf.

```python
# pip install scikit-learn
from sklearn.metrics import classification_report, confusion_matrix
pred = [classify(t)["label"] for t in texts]
print(classification_report(gold, pred, labels=LABELS, digits=3))
print(confusion_matrix(gold, pred, labels=LABELS))
```

Rerun the same set every time you change the prompt or the model. [How to evaluate an LLM before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/) covers keeping a fixed test set.

## How do you classify at volume?

Classification is input-heavy: a long system prompt, a short message and a one-word answer. Three levers matter.

**Group messages per request.** A BroadNet pay-as-you-go key allows 30 requests a minute and 4 in flight. One message per request caps you at 30 messages a minute per key, or 1,440 times that a day. Send 20 short messages as a numbered list and ask for a JSON array of 20 results: the same rate limit then covers twenty times more messages. Check that the array length matches and that each item has its number.

**Keep the prompt prefix identical.** The system prompt and examples are the same on every call. Put them first and the message last, so they can be read from cache at the cached-input price, such as $0.0435 instead of $0.175 per 1M on Qwen3.8 27B. [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/).

**Keep output short.** Output tokens cost more than input. A label and a short quote is enough; drop the evidence field once you trust the model.

Arabic usually takes more tokens per word than English, so count a sample of your messages in the [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/), then price the month in the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/).

## What mistakes should you avoid?

-   **Testing on English translations.** Translated text is not how your customers write.
-   **One score for all dialects.** Report each dialect separately.
-   **No escape label.** Forcing every message into a real label inflates errors.
-   **Free-text answers.** “It seems negative” is not parseable. Ask for JSON and validate.
-   **Ignoring sarcasm.** “ما شاء الله، ثالث مرة يتأخر الطلب” reads positive word by word and is a complaint. Put an example like it in the prompt.

## When is an LLM the wrong classifier?

When labels are stable, you already have thousands of labelled examples and volume is in the millions, a fine-tuned Arabic encoder model is usually cheaper per message and, per GPTAraEval, can be more accurate. For big offline jobs that can wait, a provider with a batch API may also cost less: OpenAI advertises a 50% discount for batch jobs completed within 24 hours. BroadNet has no batch API. A common pattern is to use an LLM to label the first few thousand messages, check them by hand, and train a small model on the result.

## Where BroadNet Inference fits

BroadNet gives you an Arabic-tested model (Qwen3.8 27B), a cheap model on the starting credit (Gemma 4 VL 26B) and a flat-rate option (DeepSeek V4.1 Flash) behind one OpenAI-compatible key, with no message content retained after the request. That suits support teams that want to classify Arabic messages without building a training pipeline. It does not suit very large offline jobs where a batch discount or a fine-tuned model would cost less. Start with the prompt above, the 200-message test, and the [Gulf, Egyptian and Levantine dialect](https://broadrouter.com/blog/arabic-llm-gulf-egyptian-dialects/) post for building the test set.

Free tools for this

-   [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) Arabic vs English token counts per tokenizer: the "Arabic tax" in tokens and money.
-   [Arabic text normalizer](https://broadrouter.com/tools/arabic-text-normalizer/) Tashkeel, tatweel, alef, ya, ta marbuta, digits and invisible characters, with a matching Python function.
-   [Speech-to-text cost calculator](https://broadrouter.com/tools/speech-to-text-cost-calculator/) Hours, calls or voice notes to a monthly transcription bill, against your current provider.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Structured output](https://broadrouter.com/glossary/structured-output/)

Structured output is an LLM API feature that constrains a model's reply to a format the developer specifies, usually a JSON Schema, so code can parse it reliably. It is stricter than JSON mode, which guarantees valid JSON but not a particular shape.

[Rate limit](https://broadrouter.com/glossary/rate-limit/)

A rate limit is a cap an API places on how much a client can use it in a time window, such as requests per minute or tokens per minute. Requests over the limit are refused, usually with HTTP 429 Too Many Requests, until usage falls back under it.

[Arabizi](https://broadrouter.com/glossary/arabizi/)

Arabizi is Arabic written informally in Latin letters, using digits for Arabic sounds with no Latin equivalent, such as 3 for ع. It is common in chat and social media, is mostly dialectal, has no standard spelling, and is often mixed with English or French.

[Batch API](https://broadrouter.com/glossary/batch-api/)

A batch API is an asynchronous LLM endpoint that accepts many requests at once, usually as a file, processes them within a set window instead of immediately, and returns the results together. Providers such as OpenAI and Anthropic price batch requests at a discount to their standard APIs.

[Modern Standard Arabic (MSA)](https://broadrouter.com/glossary/msa/)

Modern Standard Arabic (MSA) is the standardized form of Arabic used in writing, news, education and formal speech across the Arab world. It is nobody's native everyday language: people speak regional dialects such as Egyptian or Gulf Arabic and switch to MSA in formal settings.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

## Questions

### Can an LLM do Arabic sentiment analysis?

Yes, without training, by prompting it with labels and examples. Quality varies by dialect and by class: in published tests, models trail fine-tuned Arabic models on many tasks, and dialects score lower than Modern Standard Arabic. Measure it on your own data.

### Is an LLM better than a fine-tuned Arabic BERT model for classification?

Not always. GPTAraEval found ChatGPT "consistently surpassed by smaller models that have undergone finetuning on Arabic" across 44 tasks. An LLM wins when you have few labelled examples, labels that change often, or need a reason with each label.

### How do I classify Gulf and Egyptian Arabic messages?

Give the model labelled examples in each dialect, tell it which dialects to expect, and score each dialect separately. A model that does well on Egyptian reviews can still miss Gulf sarcasm.

### How do I get a consistent label every time?

Use a fixed label list, ask for JSON with only those values, set temperature to 0, and reject any reply that is not on the list. Send the same examples in the same order so results are repeatable.

### How many messages can I classify per day on BroadNet?

A pay-as-you-go key allows 30 requests a minute and 4 at once. At one message per request that is at most 30 messages a minute per key, or 1,440 times that a day. Putting 20 short messages in each request raises that twentyfold, within the context limit.

### Does BroadNet have a batch API for bulk classification?

No. BroadNet documents chat completions and transcription only. For large offline jobs, group messages per request, or compare a provider with a batch API; OpenAI's Batch API advertises a 50% discount with a 24-hour window.

## Keep reading

-   [Gulf, Egyptian and Levantine Arabic: how LLMs handle dialects](https://broadrouter.com/blog/arabic-llm-gulf-egyptian-dialects/)
-   [Arabizi and LLMs: handling Franco-Arabic with numbers like 3 and 7](https://broadrouter.com/blog/arabizi-llm-handling/)
-   [JSON mode vs json\_schema: structured output that holds up](https://broadrouter.com/blog/structured-output-json-schema/)
-   [Arabic prompt engineering: register, RTL, numerals and mixed text](https://broadrouter.com/blog/arabic-prompt-engineering/)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/)
-   [How to structure prompts so the prefix caches](https://broadrouter.com/blog/prompt-structure-for-caching/)
-   [Arabic named entity extraction with an LLM: JSON, dates and checks](https://broadrouter.com/blog/arabic-named-entity-extraction-llm/)
-   [LLM batch processing without a Batch API: a resumable runner](https://broadrouter.com/blog/llm-batch-processing-without-batch-api/)
-   [Add diacritics to Arabic text with an LLM: prompt, checks, DER](https://broadrouter.com/blog/arabic-diacritization-llm/)
-   [Arabic grammar and spelling correction with an LLM: a how-to](https://broadrouter.com/blog/arabic-grammar-spelling-correction-llm/)
-   [Guide: Why a reasoning model returns an empty answer](https://broadrouter.com/guides/reasoning-tokens-empty-response/)
-   [Guide: BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-arabic-text-classification-llm&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference: models, structured output, reasoning\_effort, pay-as-you-go limits, hourly key limits (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-arabic-text-classification-llm&utm_content=sources)
-   [BroadNet Inference pricing page: Qwen3.8 27B "Tested in Arabic" note and per-token prices (checked 2026-11-15)](https://inference.broadnet.ai/pricing?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-arabic-text-classification-llm&utm_content=sources)
-   [Alharbi et al., Evaluating Large Language Models on Sentiment Analysis in Arabic Dialects, RANLP 2025 (checked 2026-11-15)](https://aclanthology.org/2025.ranlp-1.8/)
-   [Khondaker et al., GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP (arXiv 2305.14976; checked 2026-11-15)](https://arxiv.org/abs/2305.14976)
-   [OpenAI Batch API guide: 50% discount, 24-hour window (checked 2026-11-15)](https://developers.openai.com/api/docs/guides/batch)
