---
title: "What is an LLM eval? Testing model output explained | BroadRouter"
description: "An eval is a repeatable test set that scores LLM outputs against your criteria. How to build one, how to grade it, and why to run it before switching models."
url: https://broadrouter.com/glossary/eval/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Eval (LLM evaluation)

Updated 2026-11-16

An eval (short for evaluation) is a repeatable test of a language model or prompt: a fixed set of inputs, a description of what a good output looks like, and a grader (code, a human or another model) that scores each output so that versions can be compared.

Also called: evals, LLM evaluation, model evaluation, LLM testing, eval set

An eval is a unit test for a prompt and model: same inputs every time, a clear pass mark, and a score you can compare.

## How it works

An eval has three parts:

1.  **Cases.** Inputs drawn from real use, with edge cases. Anthropic’s guide lists examples such as irrelevant input, overly long input and ambiguous cases “where even humans would find it hard to reach an assessment consensus.”
2.  **Criteria.** What a good answer looks like: an exact value, a required field, a tone, a fact that must appear.
3.  **Grader.** Something that scores each output.

| Grader | Good for | Weakness |
| --- | --- | --- |
| Code (exact match, JSON Schema, regex) | Classification, extraction, [structured output](https://broadrouter.com/glossary/structured-output/) | Cannot judge open-ended text |
| Human | Nuance, tone, new tasks | Slow and costly |
| Model ([LLM-as-judge](https://broadrouter.com/glossary/llm-as-judge/)) | Open-ended answers at scale | Has its own biases |

Anthropic recommends automating where possible and states “Prioritize volume over quality”: more cases with slightly noisier automated grading beat a few hand-graded ones. It also suggests grading with a different model from the one that produced the output.

## Why it matters

-   **Switching models or providers.** Public benchmarks rarely match your task. Even for embedding models, the MTEB benchmark found “no particular text embedding method dominates across all tasks.” Your own eval is the only reliable comparison. See [how to evaluate an LLM before switching](https://broadrouter.com/blog/evaluate-llm-before-switching/).
-   **Prompt changes.** A fix for one case often breaks another. Rerun the full set on every prompt edit.
-   **Cost trade-offs.** An eval shows whether a cheaper model is good enough for a given step, using numbers instead of impressions.
-   **Safety.** Add cases for [hallucinations](https://broadrouter.com/glossary/hallucination/), refusals and injection attempts, not only for happy paths.

## On BroadNet Inference

BroadNet’s docs do not list an evals product. Run your eval set from your own code against the OpenAI-compatible endpoint, changing only the model ID to compare models. Reasoning models count reasoning toward `max_tokens`, so give them room or a case can fail with an empty answer rather than a wrong one. The [GLM 5.3 vs DeepSeek V4.1 Flash](https://broadrouter.com/blog/glm-5-3-vs-deepseek-v4-1-flash/) post is one worked comparison.

## Questions

### What is the difference between an eval and a benchmark?

A benchmark is a public test set used to compare models in general. An eval is usually your own set, built from your real tasks, and tells you how a model does on your product.

### How many test cases does an eval need?

There is no fixed number. Anthropic's guidance favours more cases with automated grading over a few hand-graded ones; start with dozens drawn from real traffic and grow it as you find failures.

### Should I set temperature to 0 for evals?

Use the settings you will run in production. If outputs vary between runs, score each case several times and compare averages rather than single results.

## See also

-   [LLM-as-a-judge](https://broadrouter.com/glossary/llm-as-judge/) (term)
-   [Hallucination](https://broadrouter.com/glossary/hallucination/) (term)
-   [Structured output](https://broadrouter.com/glossary/structured-output/) (term)
-   [Temperature](https://broadrouter.com/glossary/temperature/) (term)
-   [Guardrails](https://broadrouter.com/glossary/guardrails/) (term)
-   [LLM benchmark](https://broadrouter.com/glossary/llm-benchmark/) (term)
-   [Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/) (guide)
-   [How to evaluate an LLM before switching providers](https://broadrouter.com/blog/evaluate-llm-before-switching/) (article)
-   [Switching to an OpenAI-compatible API: a migration checklist](https://broadrouter.com/blog/openai-compatible-api-migration-checklist/) (article)
-   [GLM 5.3 vs DeepSeek V4.1 Flash: which one to use, and when](https://broadrouter.com/blog/glm-5-3-vs-deepseek-v4-1-flash/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-eval&utm_content=page-cta) ·

**Sources**

-   [Anthropic, define success criteria and build evaluations (develop tests) (checked 2026-11-16)](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
-   [Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv) (checked 2026-11-16)](https://arxiv.org/abs/2306.05685)
-   [Muennighoff et al., MTEB: Massive Text Embedding Benchmark (arXiv) (checked 2026-11-16)](https://arxiv.org/abs/2210.07316)
