How to build an LLM judge you can trust

Using a language model to grade the output of another model is now a normal part of shipping AI. You see it in eval pipelines that score a thousand answers a night, in support systems that decide whether a conversation was resolved, in first-pass content moderation, and in quality scoring of agent transcripts. The appeal is obvious: grading is the bottleneck, and a model grades at a volume no human team can reach.

The part that usually goes unmeasured is the judge itself. A judge built from a language model has the same failure modes as the thing it grades. It can be inconsistent, it can be confidently wrong, and it can drift when the underlying model changes. A number that comes out of an unmeasured judge looks like evidence and is not.

The useful way to hold it is that the judge is an instrument. You would not report a temperature from a thermometer you never checked against a known reference. The discipline below is how you check the instrument, and how you build it so that checking is even possible.

It applies wherever you grade with a model, and the customer-support case, which I cover end to end in deploying a customer-support AI agent to production, is one worked example of it.

The judge is an instrument, so treat it like one

Start from what goes wrong when a raw model is the whole grader.

It is not reproducible. Ask the same model to grade the same case twice and the verdict can flip, because sampling is stochastic and the prompt leaves room. An eval number that changes when nothing changed is not a measurement.

It keeps no audit trail. A single “fail” with no record of why cannot be defended to anyone who asks, and in a regulated setting someone will ask.

It drifts. The vendor updates the model between Friday and Monday, the rubric gets a tweak, and yesterday’s numbers are no longer comparable to today’s, silently.

And it can penalise intended behaviour. If the judge does not know what the system was designed to do, it grades a deliberate, correct refusal as a failure to answer.

None of these are reasons to avoid an LLM judge. They are the specification for building one you can trust. Four moves handle them: settle the clear cases in code, judge only the rest, store every verdict with its provenance, and calibrate against humans.

Move 1: deterministic gates carry the clear cases

The first mistake is to treat every case as a judgement call. Whether an output is empty, whether a hard rule was violated, whether a support ticket was reopened within the hour, whether a required field is missing: these are facts you can check with code. A model is not required to decide them.

So put a layer of deterministic gates in front of the model. Each gate is a plain rule that can settle a case on its own. An empty completion fails. A response that leaks a forbidden string fails. A ticket the customer reopened is not resolved. A gate that fires ends the grading for that case, with a reason attached.

Two things make this worth doing before anything else. The gates are reproducible and free, so the cases they settle carry no model cost and never flip between runs. And they shrink the population the model has to judge, which leaves the expensive, fallible part of the pipeline pointed only at the cases that genuinely need judgement.

flowchart TD
    C["A case to grade"] --> G{"Deterministic gates<br/>empty? rule broken?<br/>reopened? missing field?"}
    G -->|a gate fires| V1["Verdict, from the gate<br/>reproducible, free"]
    G -->|nothing fires| CA{"Seen this exact<br/>case before?"}
    CA -->|yes| V2["Cached verdict<br/>identical to last run"]
    CA -->|no| J["LLM judge grades<br/>the ambiguous remainder"]
    J --> V3["Verdict, from the model<br/>with a reason"]
    V1 --> P["Store verdict + provenance"]
    V2 --> P
    V3 --> P

Step a case through the cascade below. Toggle what is true about it, and watch which stage settles the verdict and what that costs.

Move 2: the model judges only the remainder, against a written policy

What survives the gates is the genuinely ambiguous set: the cases where deciding needs reading and judgement. This is where the model earns its place. Give it the case, a rubric that defines the labels, and the policy the output was supposed to follow, and have it return a verdict plus a short reason.

The policy is the part that people skip, and skipping it introduces a bias you cannot calibrate away. A judge that does not know the system was designed to deflect a certain request will grade that deflection as a failure to help.

The error is directional. It pushes the failure count up every time, so the raw number overstates the problem until someone reconciles it. Writing down the intended behaviour and handing it to the judge is what stops the instrument from reading high by construction.

Keep the reason. A verdict with a one-line justification is the difference between a score you can audit and a score you have to take on faith. When a human later reconciles the flagged cases, the reason is where they start.

Move 3: cache the verdict, and stamp it with provenance

A judge you can trust is a judge whose numbers you can reproduce. The way to get there is to make each verdict a stored record rather than a fresh model call every time.

Key each verdict on the exact inputs that produced it: the case, the rubric version, and the model version. If those are unchanged, the verdict is read from the cache rather than regenerated, which means a re-run of the whole grading job returns identical numbers and costs almost nothing.

When one of them does change, the key changes, and only the affected cases are re-judged. This is the same idea as a build cache, applied to grading.

Store provenance alongside the verdict: which stage decided it (a gate or the model), which gate or rubric version, which model, and when. Provenance is what turns a bare score into an audit trail.

It lets you answer “why did this case fail” months later, and it lets you separate a change in the world from a change in the instrument, because you can see whether last quarter’s numbers came from the same rubric and model as this quarter’s.

Here is the shape of what gets stored, with the details invented to show the form.

Case	Deciding stage	Verdict	Reason	Rubric	Model
c-1041	gate: reopened	not resolved	ticket reopened in 40 min	r7	deterministic
c-1042	gate: empty	not resolved	no reply captured	r7	deterministic
c-1043	judge	resolved	answered the question, customer confirmed	r7	judge-2026-06
c-1044	judge	not resolved	correct deflection per policy, no solve	r7	judge-2026-06

The two gate rows never call the model and never change between runs. The two judge rows carry a reason and the model version, so a later reader can reconcile them and a later run can reuse them.

Move 4: calibrate against humans with kappa

A judge is only trustworthy to the degree it agrees with a human doing the same task carefully. So measure that agreement directly. Hand-label a sample yourself to make a gold set, run the judge over the same sample, and compare.

The instinct is to report accuracy, the share of cases where the judge matched the human. Accuracy lies when one label dominates. If 90% of conversations really are resolved, a judge that says “resolved” every single time scores 90% accuracy while measuring nothing at all. You want a number that corrects for the agreement you would get by chance.

That number is Cohen’s kappa, the standard measure of inter-rater reliability:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

Here $p_o$ is the observed agreement, the share of cases the judge and the human labelled the same, and $p_e$ is the agreement expected by chance given how often each label appears. Subtracting the chance agreement is what makes kappa honest. It runs from 1 for perfect agreement, through 0 for no better than chance, and below zero for worse than chance.

On the “always resolved” judge above, $p_o$ is high while $p_e$ is nearly as high, so kappa collapses toward zero, which is the truth the accuracy number hid.

How high is high enough depends on the stakes, and a common reading treats 0.6 to 0.8 as substantial agreement and above 0.8 as near-perfect.¹ Pick the threshold before you look at the result, so it stays a bar rather than becoming a rationalisation. When more than two people label the gold set, Fleiss’ kappa extends the same idea to many raters.

flowchart LR
    GS["Humans label<br/>a gold set"] --> RUN["Run the judge<br/>on the same set"]
    RUN --> K["Compute kappa<br/>vs the humans"]
    K --> T{"kappa above<br/>the threshold?"}
    T -->|yes| SHIP["Trust the judge<br/>on the full volume"]
    T -->|no| FIX["Fix the rubric<br/>or the prompt"]
    FIX --> RUN
    SHIP -.->|rubric or model changes| GS

Calibration is not a one-time gate. The dashed line matters: any change to the rubric or the model invalidates the last kappa, so you re-label enough of a fresh sample to re-measure. A judge that was calibrated a year ago and never rechecked is back to being an unmeasured instrument.

Reconcile against the number you are measured on

The last step connects the judge’s output to a number someone outside the team already trusts. If a vendor reports its own resolution rate, or you are billed on a count, put the judge’s number next to it and explain the gap.

The gap is usually not fraud. It is a definition difference (a conversation that merely ended versus one that was solved) or a timing difference (a verified outcome that settles a few days later).

Reconciling makes the judge’s number legible to finance, which is the point of measuring at all. That connection to a real, owned number is the same discipline I apply to spend in running AI enablement as a P&L and to cost attribution in attributing AI spend.

Key takeaways

• An LLM judge is an instrument. A number from an unmeasured instrument is not evidence, so build the judge to be checkable and then check it.
• Put deterministic gates first. Empty outputs, rule violations, and reopened tickets are facts you can check in code, and settling them there is reproducible, free, and shrinks the model’s workload.
• Judge only the ambiguous remainder, and grade it against a written policy so the judge does not penalise behaviour that was intended.
• Cache each verdict keyed on the case, rubric version, and model version, and store provenance with it, so re-runs are identical and every verdict has an audit trail.
• Calibrate against a human gold set with Cohen’s kappa rather than raw accuracy, because accuracy flatters a judge when one label dominates. Re-calibrate whenever the rubric or model changes.
• Reconcile the judge’s number against any external number you are billed or measured on, so the result is legible to the people who fund the work.

The full worked example, applied to a vendor-hosted support agent in a regulated setting, is in deploying a customer-support AI agent to production. If you are standing up an LLM judge and want a second pair of eyes on the calibration before you trust its numbers, a short call is a good place to start.

Footnotes

1. These bands are a convention, the Landis and Koch labels, so a high-stakes moderation judge may warrant a stricter bar than a low-stakes eval one. Set the threshold from the cost of a wrong verdict in your case rather than copying the numbers. ↩