Fine-tuning is often the first fix people reach for when a model gets something wrong, and often the wrong one. It costs data collection, labelling, training runs and a model you now have to maintain. Many failures disappear with a clearer prompt, retrieval of the right document, or a code check on the output. This post is about deciding which case you are in, and what to prepare if you really do need a custom model.
What problem are you trying to solve?
Name the failure before you name the fix. Most LLM failures fall into a few patterns, and each has a cheaper first step:
| Symptom | Try first | Fine-tuning helps when |
|---|---|---|
| Wrong or outdated facts | Retrieval (RAG) over your documents | Rarely; facts that change need retrieval anyway |
| Wrong format, broken JSON | Structured output, schema checks, one retry | The format is unusual and examples in the prompt are not enough |
| Inconsistent labels or categories | Clear definitions and few-shot examples | The label set is large or subtle and examples don’t fit in the prompt |
| Wrong tone or style | A style guide in the system prompt, with examples | A house style must hold across thousands of outputs |
| Domain words misheard or misspelt | A vocabulary list in the prompt, post-correction in code | Speech models mishear the same names, codes or numbers repeatedly |
| Too slow or too costly at scale | A smaller model, prompt caching, shorter prompts | A small fine-tuned model can match a large one on a narrow task |
| Wrong language or dialect | Instructions in the right language; see Arabic prompt engineering | The model understands but won’t write in the dialect you need |
OpenAI’s own guide describes the same order: write evals first, then iterate on prompts, and fine-tune for some use cases. It says the prompt engineering process “may be all you need in order to get great results.” It lists what fine-tuning adds over prompting: more examples than fit in one prompt, shorter prompts that save tokens, training on data you don’t want to send in every request, and making a smaller model good at one task.
How do you prove the failure is recurring?
With an eval set. Collect 100 to 200 real inputs with known good outputs, as in evaluating an LLM before switching. Score your current prompt. Then label every failure with a short category: wrong_label, missing_field, dialect, hallucinated_fact, and so on.
A pattern is a fine-tuning candidate only if all three are true:
- It is frequent. It accounts for a meaningful share of failures, not one or two odd cases.
- It survives the cheap fixes. You tried clearer instructions, examples, retrieval and validation, and the category is still there.
- It is consistent. People agree on the right answer. If two of your own reviewers label the same case differently, a model trained on their labels will learn the disagreement.
Write the result down as a short statement, such as this made-up example: “On our 200-case Arabic support set, the best prompt mislabels 18% of billing tickets as delivery; few-shot examples cut it to 15%.” That sentence is the brief for a fine-tune, and the number to beat.
Which simpler changes should you test first?
Run them in order of cost, and re-score after each one:
- Instructions. Define every label or field. Say what to do when unsure.
- Examples. Add three to ten real examples, including the hard cases.
- Retrieval. If answers need facts, retrieve the relevant passage. Arabic RAG search and long context vs RAG cover the choices.
- Constraints. Use structured output where the model supports it, and check the output in code. Re-ask once with the error.
- Another model. A different base model may already do what you need. The same eval set tells you in an hour.
Each step costs hours, not weeks, and each leaves you with a better baseline for a fine-tune to beat if you still need one.
What data do you need, and do you have the right to use it?
Before quality, check rights. Training data often comes from customer conversations, documents or call recordings. You need a legal basis and, often, consent or a contract term that covers training. Check your own privacy notice and customer contracts, and remove personal data you don’t need. Don’t assume your API provider is collecting it for you: BroadNet’s privacy statement, for example, says it does not retain inference content and does not use inference data to train, fine-tune or evaluate models.
Then check quality. This script reads a training file and an eval file in the common chat JSONL format (one {"messages": [...]} per line, ending with the assistant’s answer) and reports problems that quietly ruin fine-tunes: broken rows, duplicates, the same input with different answers, eval items that leaked into training, label balance and possible personal data.
import json, re, sys
from collections import Counter
DIAC = re.compile(r"[ً-ٰٟـ]") # tashkeel, dagger alef, tatweel
PII = {"email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"),
"phone": re.compile(r"(?:\+|00)\d[\d\s-]{7,}\d")}
def norm(s: str) -> str:
s = DIAC.sub("", s.lower())
s = re.sub("[إأآٱ]", "ا", s).replace("ى", "ي")
return " ".join(s.split())
def load(path):
rows, errors = [], []
for i, line in enumerate(open(path, encoding="utf-8"), 1):
try:
ex = json.loads(line)
roles = [m["role"] for m in ex["messages"]]
if roles[-1] != "assistant" or "user" not in roles:
raise ValueError(f"roles {roles}")
rows.append(ex)
except (ValueError, KeyError, IndexError, TypeError) as e:
errors.append(f"line {i}: {e}")
return rows, errors
def key(ex): # the input side, normalised: what the model sees
return norm(" ".join(m["content"] for m in ex["messages"] if m["role"] != "assistant"))
def check(train_path, eval_path):
train, errs = load(train_path)
evals, eval_errs = load(eval_path)
keys = Counter(key(e) for e in train)
labels = Counter(e["messages"][-1]["content"].strip() for e in train)
leaked = {key(e) for e in evals} & set(keys)
pii = Counter(k for e in train for m in e["messages"] for k, rx in PII.items() if rx.search(m["content"]))
conflicts = Counter()
seen = {}
for e in train:
k, a = key(e), e["messages"][-1]["content"].strip()
if k in seen and seen[k] != a:
conflicts[k] += 1
seen.setdefault(k, a)
return {"train_examples": len(train), "parse_errors": errs + eval_errs,
"duplicate_inputs": sum(n - 1 for n in keys.values() if n > 1),
"same_input_different_answer": len(conflicts),
"eval_items_also_in_train": len(leaked),
"answer_distribution": labels.most_common(5),
"examples_with_possible_pii": dict(pii)}
if __name__ == "__main__":
print(json.dumps(check(sys.argv[1], sys.argv[2]), ensure_ascii=False, indent=1))
We ran it on a seven-line toy file of Arabic, English and Spanish support tickets. It found one row without an assistant answer, two duplicate inputs (one differed only by a diacritic, which the normaliser removes, as in Arabic text normalization), one input labelled both billing and other, one eval item that also appeared in training, and one phone number. Every one of those would have made the fine-tune look better or worse than it is.
The normalisation matters for Arabic. Without it, “متأخر” and “متأخِّر” count as different inputs, and a near-copy of a test item can slip into training.
How do you evaluate a fine-tune fairly?
Use the same held-out test set you used for the baseline, and keep it out of training. Then compare:
- Against the best prompt, not the first one. A fine-tune that beats a weak prompt proves nothing.
- Per category and per language. An overall gain can hide a drop in Arabic while English improves. Report each.
- On general behaviour. Add 20 to 50 ordinary requests the model handled well before. Narrow training can degrade behaviour outside the task.
- On cost and latency. A fine-tune is often justified by a smaller model doing the job. Measure tokens and time per request, too.
Decide the pass rule before you see the numbers, as in the eval post: for example, “at least 5 points better on billing labels, no category worse by more than 2 points, Arabic and English both pass.”
What changes when you can’t fine-tune hosted models?
Fewer hosted APIs offer self-serve fine-tuning than you might expect. OpenAI’s deprecations page records an update to its self-serve fine-tuning platform: from 7 May 2026, organisations that had not run fine-tuning can no longer create jobs; from 2 July 2026, neither can organisations that had not run inference on a fine-tuned model in the past 60 days; and from 6 January 2027 active existing customers can no longer create new fine-tuning jobs. Existing fine-tuned models stay available for inference until their base models are deprecated.
The usual alternative is an open-weight model trained with a parameter-efficient method such as LoRA. The LoRA paper (Hu et al.) reports that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. You then need somewhere to serve the result, which is either your own hardware or a provider that hosts custom weights. Open-weight vs closed models covers that trade-off.
Where BroadNet Inference fits
BroadRouter is BroadNet’s own site. BroadNet Inference’s self-serve API has no fine-tuning endpoint: its API reference documents chat completions and audio transcription only. The same page says “private endpoints, our Arabic fine-tunes and other models on request are reached with a private ID issued with the endpoint.” BroadNet’s model pages list a BroadNet fine-tune of Cohere Transcribe Arabic for numbers, dates and names in noisy audio as on request, with a private model ID per customer, and say Gemma 4 VL 26B is available on a private endpoint “including our Arabic fine-tunes”. These are enquiries through the contact page, not something you can buy or train yourself; planning a private endpoint covers what to send.
For everything before that step, the public models are enough to build your eval set and baseline: Qwen3.8 27B and GLM 5.3 document structured output, Gemma 4 VL 26B runs on the $5 starting credit, and BroadNet Transcribe Arabic handles speech at $0.0042 per minute. Inference content is not retained; necessary account, billing and security metadata is retained as described in BroadNet’s privacy statement.
