---
title: "Do you need to fine-tune an LLM? A readiness checklist"
description: "When prompting, RAG or evaluation fixes the problem and fine-tuning is not needed, what training data you need, and how to test a fine-tune before you ask for one."
url: https://broadrouter.com/blog/fine-tuning-readiness/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Before fine-tuning: define the problem to solve

By the BroadNet Inference team · 7 min read · Published 2026-11-15 · Updated 2026-11-16

Fine-tune only after an evaluation shows a recurring failure that better prompts, examples, retrieval and validation did not fix. Missing knowledge usually needs retrieval, and format errors need structured output and checks. Before asking for a custom model, collect labelled examples you may legally use, hold out a test set, and record the score to beat.

## Key takeaways

-   Start with an eval set. Without a baseline score, you cannot tell whether a fine-tune helped.
-   Missing or changing facts are a retrieval problem. Fine-tuning teaches behaviour and style more reliably than it teaches facts that change.
-   OpenAI's own guide puts evals and prompt engineering before fine-tuning, and says prompting "may be all you need".
-   Training data needs rights, consistency and a held-out test set that never appears in training. A short script catches duplicates, contradictions and leaks.
-   BroadNet's self-serve API has no fine-tuning endpoint. Its Arabic fine-tunes are offered on request, on a private endpoint with a private model ID.

**On this page**

1.  What problem are you trying to solve?
2.  How do you prove the failure is recurring?
3.  Which simpler changes should you test first?
4.  What data do you need, and do you have the right to use it?
5.  How do you evaluate a fine-tune fairly?
6.  What changes when you can’t fine-tune hosted models?
7.  Where BroadNet Inference fits

Fine-tuning is often the first fix people reach for when a model gets something wrong, and often the wrong one. It costs data collection, labelling, training runs and a model you now have to maintain. Many failures disappear with a clearer prompt, retrieval of the right document, or a code check on the output. This post is about deciding which case you are in, and what to prepare if you really do need a custom model.

## What problem are you trying to solve?

Name the failure before you name the fix. Most LLM failures fall into a few patterns, and each has a cheaper first step:

| Symptom | Try first | Fine-tuning helps when |
| --- | --- | --- |
| Wrong or outdated facts | [Retrieval (RAG)](https://broadrouter.com/glossary/rag/) over your documents | Rarely; facts that change need retrieval anyway |
| Wrong format, broken JSON | [Structured output](https://broadrouter.com/glossary/structured-output/), schema checks, one retry | The format is unusual and examples in the prompt are not enough |
| Inconsistent labels or categories | Clear definitions and [few-shot examples](https://broadrouter.com/glossary/few-shot-prompting/) | The label set is large or subtle and examples don’t fit in the prompt |
| Wrong tone or style | A style guide in the [system prompt](https://broadrouter.com/glossary/system-prompt/), with examples | A house style must hold across thousands of outputs |
| Domain words misheard or misspelt | A vocabulary list in the prompt, post-correction in code | Speech models mishear the same names, codes or numbers repeatedly |
| Too slow or too costly at scale | A smaller model, [prompt caching](https://broadrouter.com/blog/prompt-structure-for-caching/), shorter prompts | A small fine-tuned model can match a large one on a narrow task |
| Wrong language or dialect | Instructions in the right language; see [Arabic prompt engineering](https://broadrouter.com/blog/arabic-prompt-engineering/) | The model understands but won’t write in the dialect you need |

OpenAI’s own guide describes the same order: write evals first, then iterate on prompts, and fine-tune for some use cases. It says the prompt engineering process “may be all you need in order to get great results.” It lists what fine-tuning adds over prompting: more examples than fit in one prompt, shorter prompts that save tokens, training on data you don’t want to send in every request, and making a smaller model good at one task.

## How do you prove the failure is recurring?

With an eval set. Collect 100 to 200 real inputs with known good outputs, as in [evaluating an LLM before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/). Score your current prompt. Then label every failure with a short category: `wrong_label`, `missing_field`, `dialect`, `hallucinated_fact`, and so on.

A pattern is a fine-tuning candidate only if all three are true:

1.  **It is frequent.** It accounts for a meaningful share of failures, not one or two odd cases.
2.  **It survives the cheap fixes.** You tried clearer instructions, examples, retrieval and validation, and the category is still there.
3.  **It is consistent.** People agree on the right answer. If two of your own reviewers label the same case differently, a model trained on their labels will learn the disagreement.

Write the result down as a short statement, such as this made-up example: “On our 200-case Arabic support set, the best prompt mislabels 18% of billing tickets as delivery; few-shot examples cut it to 15%.” That sentence is the brief for a fine-tune, and the number to beat.

## Which simpler changes should you test first?

Run them in order of cost, and re-score after each one:

1.  **Instructions.** Define every label or field. Say what to do when unsure.
2.  **Examples.** Add three to ten real examples, including the hard cases.
3.  **Retrieval.** If answers need facts, retrieve the relevant passage. [Arabic RAG search](https://broadrouter.com/blog/arabic-rag-search/) and [long context vs RAG](https://broadrouter.com/blog/long-context-vs-rag/) cover the choices.
4.  **Constraints.** Use structured output where the model supports it, and check the output in code. Re-ask once with the error.
5.  **Another model.** A different base model may already do what you need. The same eval set tells you in an hour.

Each step costs hours, not weeks, and each leaves you with a better baseline for a fine-tune to beat if you still need one.

## What data do you need, and do you have the right to use it?

Before quality, check rights. Training data often comes from customer conversations, documents or call recordings. You need a legal basis and, often, consent or a contract term that covers training. Check your own privacy notice and customer contracts, and remove personal data you don’t need. Don’t assume your API provider is collecting it for you: BroadNet’s privacy statement, for example, says it does not retain inference content and does not use inference data to train, fine-tune or evaluate models.

Then check quality. This script reads a training file and an eval file in the common chat JSONL format (one `{"messages": [...]}` per line, ending with the assistant’s answer) and reports problems that quietly ruin fine-tunes: broken rows, duplicates, the same input with different answers, eval items that leaked into training, label balance and possible personal data.

```python
import json, re, sys
from collections import Counter

DIAC = re.compile(r"[ً-ٰٟـ]")          # tashkeel, dagger alef, tatweel
PII = {"email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"),
       "phone": re.compile(r"(?:\+|00)\d[\d\s-]{7,}\d")}

def norm(s: str) -> str:
    s = DIAC.sub("", s.lower())
    s = re.sub("[إأآٱ]", "ا", s).replace("ى", "ي")
    return " ".join(s.split())

def load(path):
    rows, errors = [], []
    for i, line in enumerate(open(path, encoding="utf-8"), 1):
        try:
            ex = json.loads(line)
            roles = [m["role"] for m in ex["messages"]]
            if roles[-1] != "assistant" or "user" not in roles:
                raise ValueError(f"roles {roles}")
            rows.append(ex)
        except (ValueError, KeyError, IndexError, TypeError) as e:
            errors.append(f"line {i}: {e}")
    return rows, errors

def key(ex):     # the input side, normalised: what the model sees
    return norm(" ".join(m["content"] for m in ex["messages"] if m["role"] != "assistant"))

def check(train_path, eval_path):
    train, errs = load(train_path)
    evals, eval_errs = load(eval_path)
    keys = Counter(key(e) for e in train)
    labels = Counter(e["messages"][-1]["content"].strip() for e in train)
    leaked = {key(e) for e in evals} & set(keys)
    pii = Counter(k for e in train for m in e["messages"] for k, rx in PII.items() if rx.search(m["content"]))
    conflicts = Counter()
    seen = {}
    for e in train:
        k, a = key(e), e["messages"][-1]["content"].strip()
        if k in seen and seen[k] != a:
            conflicts[k] += 1
        seen.setdefault(k, a)
    return {"train_examples": len(train), "parse_errors": errs + eval_errs,
            "duplicate_inputs": sum(n - 1 for n in keys.values() if n > 1),
            "same_input_different_answer": len(conflicts),
            "eval_items_also_in_train": len(leaked),
            "answer_distribution": labels.most_common(5),
            "examples_with_possible_pii": dict(pii)}

if __name__ == "__main__":
    print(json.dumps(check(sys.argv[1], sys.argv[2]), ensure_ascii=False, indent=1))
```

We ran it on a seven-line toy file of Arabic, English and Spanish support tickets. It found one row without an assistant answer, two duplicate inputs (one differed only by a diacritic, which the normaliser removes, as in [Arabic text normalization](https://broadrouter.com/blog/arabic-text-normalization-search/)), one input labelled both `billing` and `other`, one eval item that also appeared in training, and one phone number. Every one of those would have made the fine-tune look better or worse than it is.

The normalisation matters for Arabic. Without it, “متأخر” and “متأخِّر” count as different inputs, and a near-copy of a test item can slip into training.

## How do you evaluate a fine-tune fairly?

Use the same held-out test set you used for the baseline, and keep it out of training. Then compare:

-   **Against the best prompt, not the first one.** A fine-tune that beats a weak prompt proves nothing.
-   **Per category and per language.** An overall gain can hide a drop in Arabic while English improves. Report each.
-   **On general behaviour.** Add 20 to 50 ordinary requests the model handled well before. Narrow training can degrade behaviour outside the task.
-   **On cost and latency.** A fine-tune is often justified by a smaller model doing the job. Measure tokens and time per request, too.

Decide the pass rule before you see the numbers, as in the eval post: for example, “at least 5 points better on billing labels, no category worse by more than 2 points, Arabic and English both pass.”

## What changes when you can’t fine-tune hosted models?

Fewer hosted APIs offer self-serve fine-tuning than you might expect. OpenAI’s deprecations page records an update to its self-serve fine-tuning platform: from 7 May 2026, organisations that had not run fine-tuning can no longer create jobs; from 2 July 2026, neither can organisations that had not run inference on a fine-tuned model in the past 60 days; and from 6 January 2027 active existing customers can no longer create new fine-tuning jobs. Existing fine-tuned models stay available for inference until their base models are deprecated.

The usual alternative is an [open-weight model](https://broadrouter.com/glossary/open-weight-model/) trained with a parameter-efficient method such as [LoRA](https://broadrouter.com/glossary/lora/). The LoRA paper (Hu et al.) reports that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. You then need somewhere to serve the result, which is either your own hardware or a provider that hosts custom weights. [Open-weight vs closed models](https://broadrouter.com/blog/open-weight-vs-closed-models/) covers that trade-off.

## Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference’s self-serve API has no fine-tuning endpoint: its [API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-fine-tuning-readiness&utm_content=body) documents chat completions and audio transcription only. The same page says “private endpoints, our Arabic fine-tunes and other models on request are reached with a private ID issued with the endpoint.” BroadNet’s model pages list a BroadNet fine-tune of Cohere Transcribe Arabic for numbers, dates and names in noisy audio as on request, with a private model ID per customer, and say Gemma 4 VL 26B is available on a private endpoint “including our Arabic fine-tunes”. These are enquiries through the [contact page](https://broadrouter.com/contact/#private), not something you can buy or train yourself; [planning a private endpoint](https://broadrouter.com/blog/private-endpoint-planning/) covers what to send.

For everything before that step, the public models are enough to build your eval set and baseline: [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) and [GLM 5.3](https://broadrouter.com/models/glm-5.3/) document structured output, [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) runs on the $5 starting credit, and [BroadNet Transcribe Arabic](https://broadrouter.com/models/cohere-transcribe-arabic/) handles speech at $0.0042 per minute. Inference content is not retained; necessary account, billing and security metadata is retained as described in BroadNet’s privacy statement.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Fine-tuning](https://broadrouter.com/glossary/fine-tuning/)

Fine-tuning is the process of continuing to train a pre-trained language model on a smaller, task-specific dataset, such as example conversations, so that its weights change and it performs that task or style more reliably without needing long instructions or examples in every prompt.

[Eval (LLM evaluation)](https://broadrouter.com/glossary/eval/)

An eval (short for evaluation) is a repeatable test of a language model or prompt: a fixed set of inputs, a description of what a good output looks like, and a grader (code, a human or another model) that scores each output so that versions can be compared.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Structured output](https://broadrouter.com/glossary/structured-output/)

Structured output is an LLM API feature that constrains a model's reply to a format the developer specifies, usually a JSON Schema, so code can parse it reliably. It is stricter than JSON mode, which guarantees valid JSON but not a particular shape.

[Prompt engineering](https://broadrouter.com/glossary/prompt-engineering/)

Prompt engineering is the process of writing, structuring and testing the instructions given to a language model so that it consistently produces output that meets your requirements. It covers role messages, clear instructions, examples, formatting, and measuring results against test cases.

[Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/)

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

[Chat Completions API](https://broadrouter.com/glossary/chat-completions-api/)

The Chat Completions API is OpenAI's HTTP endpoint, POST /v1/chat/completions, that takes a model name and a list of role-tagged messages and returns the model's next message. Its request and response shape became the common format that OpenAI-compatible providers and SDKs reuse.

## Questions

### Should I fine-tune or use RAG?

Use retrieval when the model lacks facts, especially facts that change: documents, prices, policies. Consider fine-tuning when the model has the information but behaves wrongly in a consistent way, such as format, tone, labels or domain vocabulary, and prompts with examples did not fix it.

### How do I know if prompt engineering is enough?

Measure it. Build an eval set from real inputs, score the current prompt, then try clearer instructions, few-shot examples, structured output and validation. If the score meets your target, stop. If one failure type persists across those changes, that is a fine-tuning candidate.

### How much data do I need to fine-tune?

There is no universal number. It depends on the task, the base model and the method. Collect enough labelled, consistent examples to cover every case you care about, plus a separate held-out test set, and check whether the score still rises as you add data.

### Can I fine-tune a model on BroadNet Inference?

Not self-serve. BroadNet's API reference documents chat completions and audio transcription, with no training endpoint. BroadNet lists Arabic fine-tunes, such as a tuned Cohere Transcribe Arabic, as on-request models served on a private endpoint with a private model ID.

### Does BroadNet train on my API traffic?

BroadNet's privacy statement says it does not use inference data to train, fine-tune or evaluate models, and does not retain inference content. If you want to fine-tune on your own traffic, you must collect and store that data yourself, with the right to use it.

## Keep reading

-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/)
-   [Arabic prompt engineering: register, RTL, numerals and mixed text](https://broadrouter.com/blog/arabic-prompt-engineering/)
-   [Arabic RAG: normalisation, chunking and retrieval that works](https://broadrouter.com/blog/arabic-rag-search/)
-   [Long context vs RAG: when 1M tokens replaces a vector DB](https://broadrouter.com/blog/long-context-vs-rag/)
-   [JSON mode vs json\_schema: structured output that holds up](https://broadrouter.com/blog/structured-output-json-schema/)
-   [Open-weight vs closed AI models for business: cost, control, risk](https://broadrouter.com/blog/open-weight-vs-closed-models/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [BroadNet's GPU hardware and serving approach, as BroadNet describes it](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/)
-   [Cache LLM API responses to cut cost: an exact-match cache in Python](https://broadrouter.com/blog/cache-llm-responses-cut-cost/)
-   [Guide: Arabic speech-to-text API, step by step](https://broadrouter.com/guides/arabic-speech-to-text-api/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-fine-tuning-readiness&utm_content=page-cta) ·

**Sources**

-   [OpenAI, Model optimization guide: evals, prompt engineering and fine-tuning (checked 2026-11-15)](https://platform.openai.com/docs/guides/fine-tuning)
-   [OpenAI, Deprecations: update to self-serve fine-tuning, dated 7 May 2026, 2 July 2026 and 6 January 2027 (checked 2026-11-15)](https://platform.openai.com/docs/deprecations)
-   [Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv 2106.09685 (checked 2026-11-15)](https://arxiv.org/abs/2106.09685)
-   [BroadNet Inference API reference: endpoints, and "our Arabic fine-tunes" reached with a private ID (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-fine-tuning-readiness&utm_content=sources)
-   [BroadNet model page, Cohere Transcribe Arabic BroadNet fine-tune: on request, private ID per customer (checked 2026-11-15)](https://www.broadnet.ai/inference/models/cohere-transcribe-arabic-broadnet)
-   [BroadNet model page, Gemma 4 VL 26B: private endpoint "including our Arabic fine-tunes" (checked 2026-11-15)](https://www.broadnet.ai/inference/models/gemma-4-vl-26b)
-   [BroadNet Inference privacy statement: inference data not used to train, fine-tune or evaluate models (checked 2026-11-15)](https://www.broadnet.ai/inference-privacy)
