← All writing AI engineering, 13 min read

LLM as a judge in production: ship it in shadow mode first

LLM as a judge in production: replacing a regex quality gate with a judge that runs in shadow mode, fails open to a deterministic floor, and flips on numbers.

Every outreach sequence our agents draft at Luminik goes through a quality gate before a human sees it. For a long time the hardest part of that gate was a set of regular expressions trying to answer questions about English phrasing, like "does the first email ask for a meeting before it has given the reader a reason to care?"

LLM as a judge means using a language model to grade or classify output, often another model's output, in place of a hand-written rule or a human reviewer. Replacing those regexes with one is what this post is about.

Regexes are a poor fit for that kind of question:

  • each false positive blocked a good draft;
  • each fix added another pattern;
  • the next false positive came from a phrasing nobody had thought of.

An LLM judge is the obvious replacement. The harder question is how to put a model in the path of real output without blocking or passing the wrong drafts while you learn.

This post is how I did it, in 6 parts:

  1. what replacing one judge with another means, from first principles;
  2. splitting the gate into layers, so the judge can only change one of them;
  3. the 3 modes, and what shadow mode counts;
  4. which way the gate fails when the judge does;
  5. the 31-case eval that changed which model we would have shipped;
  6. when to flip, and what it costs.

It is the rollout half of judging. The calibration half, gold sets and Cohen's kappa, is in How to build an LLM judge you can trust.

What the gate checks

A sequence is a handful of emails a model writes for one prospect. Before it reaches the approval queue, the gate checks it in 2 ways.

  • Rules a program can check exactly. A template placeholder left in the copy, a link the sequence is not allowed to carry, a phrase on the banned list, a subject line in the wrong case. These are facts. A program answers them the same way every time, for free.
  • Questions about intent. Does the opener talk about the seller before the reader? Does the first email ask for a call too early? Is a question in the copy a leading one? These depend on meaning, and the same intent shows up in many phrasings.

The regexes were answering the second kind, and the bug list showed the strain:

  • a greeting followed by a newline read as a pronoun;
  • a question that started with "has" got missed, while the same question with "have" was caught;
  • a duration like "15 minutes" in an unrelated sentence counted as a meeting ask.

Each one was fixed with another pattern.

LLM as a judge, from first principles

Strip the problem down and there are 2 classifiers. The regexes are one, the LLM is the other. Each reads a draft and says which checks fail.

Each one can be wrong in 2 directions:

  • a false positive blocks a good draft;
  • a false negative lets a bad draft through.

Replacing one classifier with the other changes both error rates at once, on drafts nobody has seen yet. 3 requirements follow from that, and every design decision in this post comes from one of them.

  1. Measure both on the same inputs. Error rates measured on different drafts cannot be compared. So the new judge has to see the drafts the old one sees, at the same time. That is shadow mode.
  2. Limit what the new one can break while you measure. So the checks a program can do exactly stay out of the judge's reach. That is the deterministic floor.
  3. Keep a way back that does not depend on the new one. So there is a kill switch, and a fixed rule for what happens when the judge fails.

Medicine has used the same arrangement for a long time. Breast screening programs in the UK use double reading: 2 radiologists read each mammogram independently, and a third settles disagreements. Shadow mode borrows the first half of that. The judge reads every draft alongside the regexes, and the disagreements are counted and read by a person.

The 2 classifiers fail in different ways:

Regex rulesLLM judge
Cost per checkfreea fraction of a cent, more for a frontier model
Latencymicrosecondsabout 0.2 seconds and up
Same input, same answeralwaysusually, with some variation between runs
A phrasing nobody planned formissed or misreadhandled if the judge reads the intent
How it failsa wrong pattern, every timea wrong call, a timeout, a malformed answer
How you audit a verdictread the patternread the reason and the prompt version

Split the gate into layers

The first change was to the gate's structure. I split it into 3 layers so that the judge could only ever change one of them.

  1. The deterministic floor. Every rule a program can check exactly. It always enforces, in every mode, and no judge result can switch it off.
  2. The intent layer. The phrasing questions. Either the old regexes or the new judge can answer them, and a setting picks which one enforces.
  3. The shadow. Whichever of the 2 intent answerers is not enforcing still runs, and its verdict is compared with the enforcing one and logged.

The split limits what the judge can affect. Whatever the judge does, a draft with a leftover placeholder never passes, because the floor does not consult it.

In shadow mode the regexes still decide. The judge's answer is compared with theirs and counted. In enforce mode the 2 swap places, and the floor stays where it is.

Three modes and one switch

The intent layer has 3 modes, and one runtime setting chooses between them.

ModeFloorWhat enforces intentJudge call
offenforcesnothingnever runs
shadowenforcesregexesruns, compared and logged
enforceenforcesjudgeruns; regexes become the logged shadow

3 properties of that table follow from the requirements above.

  • shadow is the default. A new environment, a test run or a misconfigured deploy lands in the mode where the judge decides nothing.
  • off is the kill switch. It drops intent checks entirely and keeps the floor. Changing it is a config change on the running service, with no build and no deploy. If the judge misbehaves at 2 a.m., the fix is one value.
  • enforce keeps a shadow. After the flip the regexes keep running and keep being compared. The disagreement counter means the same thing before and after the flip, so a jump after flipping is easy to see.

Here is the orchestration, reduced to its shape. The names are made up for this post; the structure is what we run.

from collections import Counter
from dataclasses import dataclass
from typing import Literal

Mode = Literal["off", "shadow", "enforce"]
Finding = tuple[str, int]  # (finding code, email step; 0 for the whole sequence)
COUNTERS: Counter[str] = Counter()

@dataclass(frozen=True)
class GateResult:
    findings: list[Finding]
    mode: Mode
    judge_error: str | None = None
    agreement: bool | None = None

async def run_gate(seq, mode: Mode, judge, regex_rules, floor_rules) -> GateResult:
    floor = floor_rules(seq)                  # always enforced
    if mode == "off":
        return GateResult(floor, mode)

    legacy = set(regex_rules(seq))
    try:
        judged = set(await judge(seq))
    except Exception as exc:                  # timeout, bad schema, oversized input
        COUNTERS["judge_error"] += 1
        log_warning("gate judge failed", mode=mode, judge=judge.name,
                    prompt_version=judge.prompt_version, error=type(exc).__name__)
        if mode == "shadow":
            return GateResult(floor + sorted(legacy), mode, type(exc).__name__)
        return GateResult(floor, mode, type(exc).__name__)   # fail open to the floor

    agree = judged == legacy
    COUNTERS["shadow_agreement" if agree else "shadow_disagreement"] += 1
    log_info("gate shadow comparison", mode=mode,
             judge_only=sorted(judged - legacy), legacy_only=sorted(legacy - judged))

    enforcing = judged if mode == "enforce" else legacy
    return GateResult(floor + sorted(enforcing), mode, None, agree)

What shadow mode counts

A single "the judge agreed" bit hides the part you need. The comparison works on the set of findings, where each finding is a pair: which check fired, and on which email in the sequence.

For any one finding, there are 4 cases:

Regex flags itRegex passes it
Judge flags itagreementjudge only: a regex false negative, or a judge stricter than intended
Judge passes itregex only: a judge miss, or a regex false positiveagreement

The 2 kinds of disagreement need different follow-up:

  • Mostly "regex only", and a person reading a sample sides with the judge: the regexes were costing you good drafts, and the migration is paying off.
  • Mostly "judge only": the judge is the stricter reviewer, and you need to decide whether you want that.

The step number matters as well. "Meeting ask on email 1" and "meeting ask on email 3" are different verdicts. Early on we compared codes only, and 2 answerers could "agree" while pointing at different emails.

The logs carry codes and step numbers and nothing else. The copy is a prospect-facing email with a real name in it. An agreement metric does not need it, and leaving it out keeps personal data out of the logs.

Which way the gate fails

The judge is a network call to a model. It will time out, return a shape that does not match the schema, or be unavailable. I wrote down what happens in each case before writing the code.

  • No judge failure kills a job. A sequence job never errors because the judge did. The failure is logged with the mode, the judge's name and the prompt version, and counted as judge_error.
  • In shadow mode a failure changes nothing. The regexes were deciding anyway. We lose one comparison sample.
  • In enforce mode a failure falls back to the floor alone. The regexes stay off.
A judge failure never stops a sequence. What decides instead depends on the mode, and a person still approves the draft before anything is sent.

When the judge is down in enforce mode, why not fall back to the regexes, since they are still there and still running?

Because the regexes' false positives are the reason for replacing them. A fallback to them would bring back the blocked drafts the judge was meant to remove, at random times, whenever the model provider had an outage. The gate's strictness would then depend on the provider's uptime.

There are 3 options, and each one fits a different setting:

OptionDuring a judge outageThe riskPick it when
Fail closeddrafts wait in a queuework stalls until the judge is backthe judge is the last check before a customer
Fall back to the old rulethe old false positives returnstrictness changes with provider uptimethe old rule was good and you are only adding coverage
Fall back to the floorintent checks are skippedan intent problem can pass the gateanother check, here a person, sits behind the gate

I chose the third. The floor still catches every rule a program can check, and a human approves each sequence before anything is sent.

One failure needs its own rule: input that is too big. A long sequence can exceed the size we allow inside the judge's input. Truncating it would mean judging part of the copy and passing the rest unread. We treat oversized input as a judge error and fail open by the same rule, which means the floor still sees the whole thing.

Pin the prompt, and treat the copy as untrusted

2 smaller decisions made the verdicts auditable.

The judge prompt is pinned by version. Every gate result records the name and version of the prompt that produced it. When someone asks why a draft was blocked on a given day, the answer points to one exact prompt. Our other prompts can be changed at runtime through a prompt-management tool, and I excluded this one on purpose. A prompt edit there would change gate verdicts without a deploy and without an eval run. Judge prompt changes ship as a version bump with a new eval baseline, like any other code change.

The copy goes in as untrusted data. The thing being judged is model output, and model output can contain text that reads like an instruction. The judge prompt wraps the copy in a fenced block and tells the model that everything inside it is data to classify. Model output that addresses the judge directly is a known failure mode, and the fence is there to handle it.

The 31-case eval that changed the model

Shadow mode tells you how the judge behaves on live traffic. It cannot tell you who is right when the judge and the regexes disagree. For that you need labels.

We built 31 labeled cases: real-shaped sequences with the correct intent findings written down by a person, run through the full gate, with the judge in its real position. Then we ran 4 judges over them.

JudgeAgreed with labels95% Wilson interval
The cheap-tier model we had planned to use3 of 313% to 25%
A hosted intent classifier we piloted22 to 24 of 31 across runs53% to 84% (22), 60% to 89% (24)
Frontier model A24 of 3160% to 89%
Frontier model B24 of 3160% to 89%

The cheap model was the default judge in shadow mode at the time. Had we gone straight to enforce, it would have been deciding intent for every sequence while agreeing with a human on about 1 case in 10. In shadow it decided nothing, and the eval caught it.

31 cases are enough to rule the cheap model out. The other 3 intervals overlap almost completely, so the same 31 cases cannot rank them.

A Wilson interval is the range of true agreement rates that could plausibly have produced the count you saw.1 With 31 cases, 24 agreements is consistent with anything from about 60% to 89%. So the eval answers one question well and another one badly.

  • It answers "is this judge usable?" 3 of 31 and 24 of 31 do not overlap at all. The cheap model's upper bound, 25%, sits far below the frontier models' lower bound, 60%.
  • It does not answer "which good judge is best?" 22 and 24 out of 31 are the same result at this sample size. Picking between the classifier and the frontier models on accuracy alone would be reading noise. We chose on cost, latency and data handling instead.

The misses were informative too. 4 of the classifier's misses were on one question, whether the copy contains a neutral question about the reader's current process, and all 4 were the same kind of miss: the judge said yes when the label said no. The LLM judges we tried made the same mistake on those cases. When different models make the same mistake, the cause is more likely the question's wording or the label than any one model. That points you at the rubric before you look for a better judge.

When to flip

The rollout plan has 3 steps, and each one keeps the previous answerer running as a logged shadow.

  1. Shadow. The regexes enforce. The judge runs, and the agreement counters build up on live traffic. The labeled eval is the baseline any change has to beat.
  2. Enforce in staging, then production. The judge decides, the regexes become the shadow, and the same counters keep running with the roles reversed. Staging goes first, and production follows only after staging has produced the same evidence.
  3. Delete the regexes. Once enforce has run long enough with disagreement we accept, the regex intent rules are deleted, and the evidence goes in the pull request that deletes them.
The floor never moves. The regexes and the judge swap roles at step 2, and the regexes leave only once the logged comparison supports it.

The bar for step 2 should be written down before anyone looks at the counters. Otherwise the bar moves to wherever the numbers happen to land. These are the conditions I set:

  • On the labeled set, the judge agrees with the human labels at least 90% of the time. No judge we have tested reaches that yet: the best agreed on 24 of 31 cases, 77%. The set also has to grow before the bar means much, because even 28 of 31 (90%) has a lower bound of about 75%.
  • On live traffic, judge_error stays low enough that the floor-only fallback is rare.
  • A person has read a sample of the disagreements, both "judge only" and "regex only", and in the "regex only" sample the judge is right more often than the regexes.

At the time of writing the gate is still in shadow mode in every environment. We swapped the judge in staging to the classifier and kept the mode at shadow, so its verdicts are compared and logged and still decide nothing.

What it costs

The approach has 4 costs.

  • An extra model call per gate run, even in shadow mode, when the judge decides nothing. With the classifier that is a fraction of a cent and about 0.2 seconds per sequence. With a frontier model it is more of both.
  • 2 code paths for intent, kept alive until the regexes are deleted. The regex rules still need their bugs fixed while they enforce.
  • A labeled set to maintain. 31 cases was enough to make a decision. It will not stay representative as the copy changes, and someone has to own adding cases when a disagreement turns up a new pattern.
  • Patience. Shadow mode delays the benefit. The regex false positives kept blocking good drafts the whole time the judge was being measured.

Shadow mode has run since late August, so the regex false positives have had that much longer to block drafts. Had we gone straight to enforce, the cheap model would have made every intent decision in that time, and it agreed with a human 3 times in 31.

Key takeaways

  • Split the gate before you add the model. A deterministic floor that always enforces limits what any judge failure can do.
  • Ship the judge in shadow mode with a kill switch. The old rule decides, the judge is compared and counted, and turning it off is a config change.
  • Compare findings, and keep the direction. "Judge only" and "regex only" disagreements mean different things and need different follow-up.
  • Decide the failure direction on purpose. Here a judge failure falls back to the floor alone, because falling back to the old rule would bring back the false positives the change exists to remove. Fail closed if nothing stands behind the judge.
  • Pin the judge prompt and record its version on every verdict. Then any blocked draft can be traced to one exact prompt.
  • Put intervals on small evals. 31 labeled cases ruled out the model we were about to use, and they could not rank the good ones. Both facts were useful.

For the statistics behind those intervals, and how to check a model labeler against human labels with Cohen's kappa, see Evaluating LLM extraction before go-live.

Footnotes

  1. The textbook interval, p plus or minus 1.96 times the standard error, misbehaves at small n and near 0% or 100%. For 3 of 31 it gives a lower bound below zero, and for 31 of 31 it gives an interval of zero width. The Wilson interval stays inside 0 to 1 and has close to the coverage it claims at these sizes, which is why I default to it. ↩

Common questions

What is LLM as a judge?↓

It means using a language model to grade or classify output, often the output of another model, in place of a hand-written rule or a human reviewer. Typical uses are eval scoring, content checks and quality gates. The judge is a model too, so it needs measuring before its verdicts are trusted.

What is shadow mode for an LLM judge?↓

The judge runs on live traffic, but its verdict is only logged and compared with the system that currently decides. The old system keeps making every decision. You collect agreement and disagreement counts on real inputs at no risk, and you flip the judge to enforcing only when those numbers support it.

Should an LLM judge fail open or fail closed?↓

It depends on what sits under it. In this gate a deterministic floor always enforces, and a human approves every draft before it is sent, so a judge outage fails open to the floor. If nothing else stood between the model output and the customer, I would fail closed and queue the work instead.

How many labeled examples do you need to evaluate an LLM judge?↓

Here 31 cases were enough to rule a judge out and too few to rank 2 good ones. A judge that agrees 24 times in 31 has a 95% Wilson interval of about 60% to 89%. That is enough to reject a judge that agrees 3 times, and too wide to separate 22 from 24.

How do you know when to switch from shadow to enforce?↓

Decide the bar before you look at the counters: the judge's agreement with human labels and its interval, its error rate on live traffic, and a human read of a sample of the cases where the judge and the old rule disagree. Flip in staging first, keep the old rule running as the logged shadow, and only delete it once the enforce period has produced the same evidence.

What metrics are used to evaluate an LLM as a judge?↓

Agreement with human labels on a labeled set, reported as percent agreement and Cohen's kappa, with a confidence interval because labeled sets are small. Split disagreements by direction, false positives against false negatives, because each one costs something different. On live traffic, add the judge's error and timeout rate, its latency and its cost per call.

Keep reading

Share this
← More writing Get in touch