An eval is a unit test for a prompt and model: same inputs every time, a clear pass mark, and a score you can compare.
How it works
An eval has three parts:
- Cases. Inputs drawn from real use, with edge cases. Anthropic’s guide lists examples such as irrelevant input, overly long input and ambiguous cases “where even humans would find it hard to reach an assessment consensus.”
- Criteria. What a good answer looks like: an exact value, a required field, a tone, a fact that must appear.
- Grader. Something that scores each output.
| Grader | Good for | Weakness |
|---|---|---|
| Code (exact match, JSON Schema, regex) | Classification, extraction, structured output | Cannot judge open-ended text |
| Human | Nuance, tone, new tasks | Slow and costly |
| Model (LLM-as-judge) | Open-ended answers at scale | Has its own biases |
Anthropic recommends automating where possible and states “Prioritize volume over quality”: more cases with slightly noisier automated grading beat a few hand-graded ones. It also suggests grading with a different model from the one that produced the output.
Why it matters
- Switching models or providers. Public benchmarks rarely match your task. Even for embedding models, the MTEB benchmark found “no particular text embedding method dominates across all tasks.” Your own eval is the only reliable comparison. See how to evaluate an LLM before switching.
- Prompt changes. A fix for one case often breaks another. Rerun the full set on every prompt edit.
- Cost trade-offs. An eval shows whether a cheaper model is good enough for a given step, using numbers instead of impressions.
- Safety. Add cases for hallucinations, refusals and injection attempts, not only for happy paths.
On BroadNet Inference
BroadNet’s docs do not list an evals product. Run your eval set from your own code against the OpenAI-compatible endpoint, changing only the model ID to compare models. Reasoning models count reasoning toward max_tokens, so give them room or a case can fail with an empty answer rather than a wrong one. The GLM 5.3 vs DeepSeek V4.1 Flash post is one worked comparison.