How to evaluate an LLM judge before trusting it

Criterion first

Judge one defined quality at a time#

Groundedness, policy compliance and task completion are different constructs. A holistic “quality” prompt lets the model change the criterion from case to case and makes disagreement impossible to diagnose.

Give the judge observable anchors for pass, partial, fail and abstain. Ask it to identify evidence before assigning a verdict, and store the rubric version with the result.

Once the criterion is fixed, a calibration set can test whether the judge applies it consistently.

Calibration set

Compare with labels that have a provenance#

Build a compact set of clear positives, clear negatives, difficult boundaries and legitimate disagreements. Labels need a written rationale and, for high-stakes criteria, review by a domain owner.

Measure per-class behavior and disagreement rather than celebrating one aggregate accuracy. A judge can look strong overall while failing the rare slice that matters most.

{"case_id":"a","human":"PASS","judge":"PASS","slice":"direct"}
{"case_id":"b","human":"HOLD","judge":"PASS","slice":"missing_evidence"}
{"case_id":"c","human":"HOLD","judge":"HOLD","slice":"adversarial"}
{"case_id":"d","human":"PASS","judge":"HOLD","slice":"verbose"}
{"case_id":"e","human":"ABSTAIN","judge":"ABSTAIN","slice":"ambiguous_boundary"}

Agreement on obvious cases is not enough; controlled perturbations expose predictable judge biases.

Adversarial validation

Test the biases the literature already names#

Run these tests on answer pairs whose meaning stays fixed. The changed feature should not decide the winner.

1. 01

   Position

   Swap candidate order; a stable verdict should not follow the slot.

2. 02

   Verbosity

   Hold meaning constant while changing length; longer must not win by default.

3. 03

   Self-preference

   Blind model identity and compare across model families.

4. 04

   Criteria drift

   Replay anchors throughout long runs and across rubric versions.

Some boundary cases will remain genuinely ambiguous, so the contract must permit disagreement and abstention.

Uncertainty contract

Make disagreement and abstention visible#

A forced answer converts missing evidence into false certainty. Give the judge an abstain path and keep its evidence separate from the final policy decision.

For important gates, use independent judgments or a small jury and inspect where they disagree. A majority vote is useful only when jurors are not copies of the same failure mode.

Because the judge is a measuring instrument, any model, prompt, rubric, or decoding change requires recalibration.

Change control

A new model is a new measuring instrument#

Model upgrades, prompt edits, rubric changes and decoding parameters can all move the verdict distribution. Pin them, record them and replay the stable calibration set before promotion.

Monitor disagreement and slice-level errors after release. Production feedback should extend the calibration set, not silently rewrite old labels.

Calibration and change control define where the judge is useful—and where a person must still decide.

Conclusion

Validate the decision boundary, not the judge’s eloquence#

An LLM judge is a measurement instrument, not an authority. Its value comes from agreement with a defined criterion across relevant cases, visible disagreement and abstention, and a known operating range—not from explanations that merely sound rigorous.

Start with one criterion and a small labeled set containing clear passes, clear failures, and boundary cases. Compare judge and human decisions, inspect systematic errors, and define when the judge must abstain. If disagreement cannot be explained, the next step is better labels or a narrower task, not more automated verdicts.