Chapter 10 · concept 5 of 6 · show all

LLM as judge, and judging the judge

Using a model to grade answers scales evaluation, and adds a new thing to test. How to write narrow rubrics, which biases to expect, and how to check a judge against human labels before trusting it.

A real question

Checking 500 answers by hand takes days. Can a model do the grading?

Yes, for narrow questions such as “is this claim in the context?”, once you have checked it against human labels. A judge is a model like any other: it has biases, and it must be tested before you trust its numbers.

The problem

Why this matters

The belief

A model judge is objective.

What really happens

Studies of LLM judges found they favour the answer shown first, longer answers, and answers in their own style (Zheng et al., MT-Bench, 2023). A judge's score is only as good as its measured agreement with people.

Why not the simpler way

Exact checks (RAG-37, RAG-42) cover numbers and codes. Many answers are text: explanations, conditions, steps. Checking those by hand does not scale past a few dozen per release. A model judge can scale it, if you can show it agrees with a careful person.

The idea

The idea in plain words

LLM as judge means prompting a model to label another model's output: supported or not, correct or not, which of two answers is better. Judging the judge means measuring how often its labels match human labels, and where it fails.

A new exam marker grades a batch that senior markers already graded. Only if they agree do they mark alone, and they get spot checks after.

How it works

Step by step

1
Ask a narrow question
Not “rate this answer 1 to 10”, but “is each claim supported by the context: yes or no?”. Give the scale with examples.
2
Compare with people
Label 50 to 100 answers by hand. Run the judge on the same answers. Count agreement per verdict, not just overall.
3
Trust it only where it agrees
Use the judge for verdicts where it matches people well. Re-check when you change the judge model or prompt.

In detail

Rubric. Binary or 3-point scales with examples. One criterion per call. Ask for a short reason before the verdict so you can audit it.

Biases (Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", arXiv 2306.05685): position bias in pairwise comparisons (swap the order and average), verbosity bias (control for length), self-enhancement (use a judge from a different model family than the one being judged where you can).

Agreement. Report agreement per class, for example the share of human NOT_SUPPORTED labels the judge also marks NOT_SUPPORTED. Overall agreement hides a judge that says SUPPORTED to everything.

Cost. One call per claim or answer. Cache by answer hash so unchanged answers are not re-judged.

The code

Try it in Python

def agreement(human: list[str], judge: list[str]) -> dict[str, float]:
    """Per-class agreement: of the items people labelled X, how many did the judge label X?"""
    out = {}
    for label in set(human):
        idx = [i for i, h in enumerate(human) if h == label]
        out[label] = sum(judge[i] == label for i in idx) / len(idx)
    return out

def pairwise(judge, question, a, b):
    """Swap order to cancel position bias; only count a win if both orders agree."""
    first, second = judge(question, a, b), judge(question, b, a)
    return "A" if (first, second) == ("A", "B") else "B" if (first, second) == ("B", "A") else "tie"

See it

What we measured

Measured · judge agreement with human labels

Planned: 50 answers from the Mini RAG Lab prompt on golden questions, labelled SUPPORTED / NOT_SUPPORTED per claim by hand, then judged by a model.

Human-labelled answersnot measured

Judge agreement on NOT_SUPPORTEDnot measured

Judge agreement on SUPPORTEDnot measured

Not measured yet. It needs model calls and a labelling pass. The method above is what we will run; no numbers are shown until then.

Try it on a real example

Two judge prompts for the same faithfulness check (written by us)

Prompt

Judge prompt

Rate the quality of this answer from 1 to 10.

Quality mixes style, length, correctness and grounding. Scores drift with answer length and wording, and two runs disagree.

Choosing

Use it when, avoid it when

Use it when

Narrow judge prompts with examples, validated on human labels
Order swapping for pairwise comparisons

Avoid it when

1 to 10 quality scores
A judge from the same model and prompt as the system under test, unchecked

What it costs · One model call per judged item, plus a one-off labelling pass of 50 to 100 items.

In production

Keep a frozen, human-labelled calibration set and re-run the judge on it whenever the judge changes. Track judge agreement as its own metric. Sample judged production answers for human review every week; disagreements become new calibration items. Never let the system being tested also be its only judge.

Watch out

Common mistakes

A strong model is a reliable judge. Reliability is measured, not assumed.
High overall agreement is enough. Check agreement on the rare class: the failures.
Judges replace human review. They reduce it; spot checks stay.

Check yourself

Remember this

Remember these

Use judges for narrow, well-defined verdicts, not overall quality.
Expect position, length and self-preference bias.
Measure agreement with human labels before using the scores.
Re-validate after changing the judge model or prompt.

Say it in an interview

01I use model judges for narrow verdicts, such as whether each claim is supported.
02I expect position, verbosity and self-preference bias, and design around them.
03Before trusting a judge I measure per-class agreement with human labels.
04Pairwise judgements are run in both orders and only count when they agree.
05I have not yet measured judge agreement on my corpus; that needs a labelling pass.

Quick quiz

Three questions

Which bias did the MT-Bench study report for LLM judges?

How do you know a judge is good enough?

Q3 · from RAG-42

What does faithfulness measure?

Saved on this device. No account needed.