LLM-as-a-judge replaces a human grader with a model call: give it the question, the answer and a rubric, and read back a score.
How it works
The method was studied in depth by Zheng et al. (2023), who built MT-Bench (multi-turn questions) and Chatbot Arena (crowdsourced head-to-head votes) to test it. Their main result: GPT-4 as a judge agreed with both expert and crowd preferences “at over 80%”, the same level as human-to-human agreement.
There are two common setups:
- Single-answer grading. The judge scores one output on a scale, for example 1 to 5, against written criteria. OpenAI’s graders guide calls this a
score_modelgrader that returns “a numeric score based on the prompt within the given range.” - Pairwise comparison. The judge sees two answers to the same question and picks the better one, or a tie.
A judge prompt usually includes the original question, the answer, a reference answer if you have one, and a rubric. Ask for a short reason before the score, and constrain the score with structured output so your code can parse it.
Why it matters
Human grading does not scale to thousands of eval cases per prompt change. A judge does, but it brings biases that Zheng et al. measured:
| Bias | What happens | Mitigation |
|---|---|---|
| Position | Prefers the first (or second) answer | Swap order and require agreement |
| Verbosity | Prefers longer answers | Rubric that rewards brevity; length caps |
| Self-enhancement | Prefers its own model’s style | Use a different model as judge |
Judges are also weak at checking maths and facts they cannot verify. For those, give the judge a reference answer, or use code-based checks. A judge that misses hallucinations gives false confidence.
On BroadNet Inference
BroadNet does not offer a hosted grading service; you write the judge call yourself against the chat completions endpoint. Judge calls are billed like any request, so a judge reading long answers adds input and output tokens per case. Estimate a run with the LLM API cost calculator. If the judge is a reasoning model, leave room in max_tokens, because reasoning counts toward it. For the wider process, see evaluate an LLM before switching.