Evaluating LLM extraction before go-live: Wilson intervals and Cohen's kappa
LLM evaluation before go-live: a stratified sample, a model labeler checked with Cohen's kappa, and precision reported with Wilson intervals.
A pipeline I run proposes changes to a catalog of public listings built from web pages. New rows, corrected dates and places, filled-in fields, and rows to withhold because they should not be there. A language model reads the pages and proposes the fields. Each run proposes thousands of changes, and none of them should go live until someone can say how often they are right.
To decide whether a model labeler can stand in for a person, I use Cohen's kappa, which measures how much 2 raters agree beyond the agreement chance alone would produce.
The tempting check is to read 50 rows, find 48 correct, and call it 96%. That number is a point estimate from a small sample. The range of true precision that could produce 48 out of 50 runs from about 87% to 99%. If the bar is 95%, the true value could sit on either side of it.
This post is the harness I built to answer that question. It covers:
- what a sample can and cannot tell you, from first principles;
- how to sample so reruns reuse labels;
- where the ground truth comes from;
- how to check a model labeler against people with Cohen's kappa;
- how to turn counts into intervals, and intervals into a go-live rule.
The statistics are about 60 lines of Python, and they are all in the post.
First principles: what a sample can tell you
An election poll asks about 1,000 people and reports a result "within 3 points". Nobody believes the poll measured the country exactly. It measured 1,000 people, and the 3 points say how far the country could plausibly be from them.
A labeled sample of extracted rows works the same way. Precision on 50 rows is a measurement with noise in it. Draw a different 50 and you would get a different count: 47 one time, 49 the next.
So the useful question is "what range of true precision could have produced this count?" That range is a confidence interval, and 3 facts about it shape everything below.
- It narrows slowly. The width shrinks with the square root of the sample size. To halve it you need about 4 times the labels.
- It is lopsided near 100%. Precision cannot go above 100%, so at high precision the interval stretches further down than up.
- It describes the sample's population only. If the sample left out a whole segment of the data, no interval covers that segment.
Here is the first fact in numbers, for a sample where 96% of rows are correct:
| Labeled rows | 95% interval for precision | Width |
|---|---|---|
| 25 | 80.5% to 99.3% | 18.8 points |
| 50 | 86.5% to 98.9% | 12.4 points |
| 100 | 90.2% to 98.4% | 8.3 points |
| 200 | 92.3% to 98.0% | 5.7 points |
| 400 | 93.6% to 97.5% | 3.9 points |
| 800 | 94.4% to 97.2% | 2.7 points |
Every step from 50 to 200 to 800 labels roughly halves the width. Labels cost money and reviewer time, so the sample size is a budget decision, and it should be made before labeling starts.
What "correct" means for extracted data
Start with the simplest version. The pipeline proposes a row with some fields: a name, a date, a city, a category, a yes or no on whether the listing is in scope at all. Precision is the share of proposed rows where a field is right.
Each output type gets its own measurement, because each one fails differently:
- New rows. Is the listing in scope, and are its name, date and place right?
- Corrections. Is the new value right, and was the old value wrong?
- Withholds. When the pipeline says "hide this row", how often was the row fine? This is a false-withhold rate, and it is the number that costs you good data.
Each field also gets 2 readings. A strict reading needs an exact match to the source. A lenient reading accepts a value that is right but less precise, like the right month with no day. Reporting both stops a debate about formatting from hiding a real error rate.
The harness at a glance
Step 1: sample so that reruns reuse labels
The sample has 2 jobs. It has to represent the data, and it has to stay the same when the data grows, so that tomorrow's rerun reuses today's labels.
Stratify first. Error rates differ by segment. Listings from one region fail in ways listings from another do not, and a single website that supplies a quarter of the rows can have its own failure that a random sample might miss. So the sample is split into strata, groups that are measured separately:
- one stratum per region;
- any website that supplies more than 25% of an output type's rows gets its own stratum;
- each stratum gets a share of the sample proportional to its size, with at least 1 row.
Then pick by hash. Inside each stratum I sort rows by sha256(seed | output type | row id) and take the lowest k. A good hash spreads ids uniformly, so the lowest k is a simple random sample. The useful property is what happens when rows are added: a new row only enters the sample if its hash is lower than the current cut-off, so almost every row picked yesterday is still picked today.
I measured the difference on a test population of 1,000 rows that grew to 1,100. With the hash, 47 of 50 sampled rows stayed the same. With Python's random.sample and the same seed, 1 row stayed the same. That is the difference between relabeling 3 rows and relabeling 49.
random.sample with a seed | Lowest hashes per stratum | |
|---|---|---|
| Rows kept when the data grows 10% | 1 of 50 | 47 of 50 |
| Same sample on a rerun with the same data | yes | yes |
| Every segment represented | only by luck | yes, by construction |
| Needs to store the sample | yes, to reuse labels | no, it can be recomputed |
import hashlib
from collections import defaultdict
def sample(rows: list[dict], n: int, seed: str) -> list[dict]:
"""Proportional stratified sample; inside a stratum, the k lowest hashes."""
strata = defaultdict(list)
for r in rows:
strata[r["stratum"]].append(r)
picked = []
for group in strata.values():
k = max(1, round(n * len(group) / len(rows)))
group.sort(key=lambda r: hashlib.sha256(f"{seed}|{r['type']}|{r['id']}".encode()).hexdigest())
picked += group[:k]
return picked
Step 2: ground truth comes from the cited page
Every proposed row cites the page it came from. The harness fetches that page again at evaluation time, through the same polite fetcher the pipeline uses: it obeys robots rules, waits between requests to the same host, and skips pages behind a login.
A separate model then labels each field against the page text. 3 choices make that labeler usable:
- It is a different model from the one that extracted the data. A model grading its own output shares its own blind spots.
- It reasons before it answers. The response schema asks for the reason first and the verdict after it. Asking for the verdict first lets the model write reasoning to fit a verdict it already gave.
- It has a call budget. The harness stops at a fixed number of labeling calls, so an evaluation cannot spend more than planned.
Some checks need no model at all. A date in the past, a host on a known low-quality list, a number formatted the European way in English copy. These are code checks. They run on every sampled row, cost nothing, and give the same answer every time.
Step 3: check the labeler against people with Cohen's kappa
With a model labeler, the precision you report is only as good as its labels. If it is wrong on 20% of rows for a field, the precision for that field is mostly measuring the labeler's errors.
So the harness writes a review sheet of 60 rows: the page excerpt, the proposed record, and blank columns for a person to fill in. Once it is filled, the harness compares the human and model labels field by field and reports 2 numbers.
The first is percent agreement. It sounds like enough, and it is misleading when one answer is common. Suppose 50 of the 60 rows are in scope. A labeler that says "in scope" every time without reading anything agrees with the person on 50 of 60 rows, 83%.
The second number fixes that. Cohen's kappa asks how much better than chance the 2 raters agree.
- Observed agreement is the share of rows where they gave the same label.
- Chance agreement is what you would expect if each rater labeled at random with their own label frequencies. If the person says "yes" 5/6 of the time and the model says "yes" 52/60 of the time, they would both say "yes" by chance on 5/6 times 52/60 of the rows, plus a similar term for "no".
- Kappa is observed minus chance, divided by 1 minus chance: the share of the possible improvement over chance that the raters achieved.
A worked example with 60 rows:
| Model: in scope | Model: out of scope | |
|---|---|---|
| Person: in scope | 48 | 2 |
| Person: out of scope | 4 | 6 |
Observed agreement is 54 of 60, 90%. Chance agreement is (50/60 × 52/60) + (10/60 × 8/60), about 74%. Kappa is (0.90 − 0.74) / (1 − 0.74), about 0.61. The always-"in scope" labeler from before has 83% agreement and a kappa of exactly 0.
The usual convention reads 0.61 to 0.80 as substantial agreement. A field where kappa comes in well below that gets its precision reported as unverified, however good the number looks.
from collections import Counter
def kappa(pairs: list[tuple[str, str]]) -> float:
"""Cohen's kappa for (human, model) label pairs."""
n = len(pairs)
observed = sum(a == b for a, b in pairs) / n
human, model = Counter(a for a, _ in pairs), Counter(b for _, b in pairs)
chance = sum(human[c] * model[c] for c in human.keys() | model.keys()) / n**2
return (observed - chance) / (1 - chance)
Step 4: precision with an interval
With trusted labels, precision for a field is k correct out of n labeled. The question is what range of true precision could have produced that count.
The textbook answer is p plus or minus 1.96 standard errors, the Wald interval. It breaks down at high precision and small samples, which is where extraction evals sit. For 50 correct out of 50, the standard error is zero, so the interval is 100% to 100%. That claims certainty from 50 rows. For 48 of 50 it gives an upper bound of 101.4%.
The Wilson interval comes from asking which true rates would make the observed count unsurprising, and it stays inside 0 to 1.1 It is a few lines:
import math
Z = 1.959964 # two-sided 95%
def wilson(k: float, n: float) -> tuple[float, float, float]:
"""Proportion k/n with its Wilson 95% interval."""
p = k / n
den = 1 + Z**2 / n
center = (p + Z**2 / (2 * n)) / den
half = Z * math.sqrt(p * (1 - p) / n + Z**2 / (4 * n**2)) / den
return p, max(0.0, center - half), min(1.0, center + half)
| Correct / labeled | Point estimate | Wald 95% | Wilson 95% |
|---|---|---|---|
| 50 / 50 | 100% | 100% to 100% | 92.9% to 100% |
| 48 / 50 | 96.0% | 90.6% to 101.4% | 86.5% to 98.9% |
| 96 / 100 | 96.0% | 92.2% to 99.8% | 90.2% to 98.4% |
| 290 / 300 | 96.7% | 94.6% to 98.7% | 94.0% to 98.2% |
The 3 common choices for a proportion compare like this:
| Method | 50 of 50 | 48 of 50 | Stays within 0 to 100% | Closed form |
|---|---|---|---|---|
| Wald (textbook) | 100% to 100% | 90.6% to 101.4% | no | yes |
| Wilson | 92.9% to 100% | 86.5% to 98.9% | yes | yes |
| Clopper-Pearson (exact) | 92.9% to 100% | 86.3% to 99.5% | yes | no, needs the beta distribution |
The first table in this step also shows what more labels buy. The same 96% point estimate has a lower bound of 86.5% at 50 rows and 90.2% at 100 rows. If your bar is a 90% lower bound, 50 rows fails and 100 rows passes, with nothing about the data changed.
Step 5: weight by stratum, and count the sample honestly
Weighting matters when strata are not sampled in proportion, for example when a small stratum gets extra rows so its own interval is usable. If the US is 90% of the population and the rest of the world is 10%, and you sample 100 US rows and 50 others, pooling the 150 labels overweights the smaller stratum.
The fix is to estimate each stratum's precision and weight it by the stratum's share of the population. Say the US stratum has 95 of 100 correct and the other has 40 of 50.
- Pooled: 135 of 150, 90%.
- Weighted by population: 0.9 × 95% + 0.1 × 80% = 93.5%.
The pooled number understates precision, because the weaker stratum got more than its share of the sample.
The interval needs one more adjustment. Unequal weights make a sample worth less than its raw count, because a few rows carry a large share of the weight. The Kish effective sample size measures that loss: the square of the sum of the weights divided by the sum of their squares. In this example the 150 labels are worth about 120. The harness computes the Wilson interval on the effective n, which gives 87.6% to 96.7%.
def stratified(cells: dict[str, tuple[int, int]], pops: dict[str, int]):
"""cells: stratum -> (correct, labeled). pops: stratum -> population size."""
total = sum(pops[s] for s in cells)
p = sum(pops[s] / total * k / n for s, (k, n) in cells.items())
var = sum((pops[s] / total) ** 2 * (k / n) * (1 - k / n) / n for s, (k, n) in cells.items())
w = [pops[s] / total / n for s, (_, n) in cells.items() for _ in range(n)]
n_eff = sum(w) ** 2 / sum(x * x for x in w) # Kish effective sample size
if 0 < p < 1 and var > 0:
n_eff = min(n_eff, p * (1 - p) / var)
_, lo, hi = wilson(p * n_eff, n_eff)
return p, lo, hi, n_eff
Step 6: compare with what is already live
A precision bar answers "is this good enough?" A second question can be more useful: "is this worse than what we already publish?" The live catalog is not perfect either, and new data that matches it may be fine to ship even if it misses an abstract bar.
So the harness labels a baseline sample of 300 rows from the published catalog with the same labeler and the same rules. For each new output type it computes the difference in precision and puts an interval on the difference using Newcombe's method, which combines the 2 Wilson intervals.
- New data 282 of 300, baseline 291 of 300: a difference of −3.0 points, with an interval of −6.6 to +0.4. The interval contains 0, so this sample cannot show the new data is worse.
- New data 270 of 300 against the same baseline: −7.0 points, interval −11.2 to −3.1. The whole interval is below 0. The new data is worse than what is live, and it does not ship.
def newcombe(k1: int, n1: int, k2: int, n2: int):
"""Difference p1 - p2 with Newcombe's hybrid score interval."""
p1, l1, u1 = wilson(k1, n1)
p2, l2, u2 = wilson(k2, n2)
d = p1 - p2
return d, d - math.hypot(p1 - l1, u2 - p2), d + math.hypot(u1 - p1, p2 - l2)
Step 7: gate on the lower bound
The last step turns numbers into a decision. I wrote the bars down before the first labeled run, per output type. For new rows, in-scope precision has 2 bars: a point estimate of at least 95%, and a lower bound of at least 90%. Some failures have a bar of zero, like a row from a known low-quality website getting through. One of those is enough to fail the type.
Each output type then gets one of 4 verdicts:
- Go live. Every bar passes, and the type is not significantly worse than the baseline.
- Fix and re-test. A bar fails or the type is worse than the baseline. Sometimes the fix is in the pipeline, and sometimes it is more labels.
- Drop. Precision is far enough below the bar that a fix is unlikely to close the gap.
- Not enough data. Fewer than 20 labeled rows. The harness refuses to judge, because an interval from 18 rows is too wide to act on.
Gating on the lower bound also changes how labeling gets done. With only a point estimate, it is easy to stop labeling once the number looks good. With a lower-bound bar, a small sample fails on its own, and the way to pass is to label enough rows.
Running it
The harness only measures. 3 properties keep it that way.
- It writes nothing outside its own folder. It reads the run's output files and a read-only snapshot of the warehouse taken before the run. It never touches the live data and never sets the flags that let a change go live. A person reads the report and makes that call.
- It is safe to rerun. Fetched pages and labels are cached, a file lock stops 2 runs overlapping, and if the inputs have not changed since the last run, it exits without doing anything.
- It can regenerate a report on a frozen population. When labeling stops partway, the report can be rebuilt on the populations the samples were drawn from, so the numbers are not mixed with rows that arrived later.
The first full run labeled 990 rows before it hit a shared model budget that other jobs also draw from. Several output types had produced nothing yet at the time the inputs were captured, so the report marked them "not checked yet".
The same rule for a duplicate check
The pattern carried over to a second job: finding duplicate listings across the whole catalog. Rules merge the clear pairs. Ambiguous pairs go to a model, which returns structured answers (same brand, same dates, same place) and an overall verdict.
2 rules from the harness applied there too:
- A verdict that contradicts its own answers counts as unsure. If the model says "same listing" while answering "different dates", neither answer is trusted, and the pair goes to review.
- Nothing is applied until a labeled sample shows merge precision of at least 98%. A wrong merge hides a real listing, so the bar is higher than for new rows.
What this does not measure
The harness measures precision on what the pipeline proposed. It leaves 3 questions open.
- Recall. Listings the pipeline never proposed are invisible to it. Measuring recall needs a sample drawn from the world, such as a list of listings you know should exist, and that is a separate exercise.
- Page drift. The page is fetched at evaluation time, and it may have changed since extraction. A correct extraction from last week's page can be labeled wrong today. For fast-changing fields, compare against a page snapshot from extraction time.
- Shared blind spots. A model labeler and a person can agree and both be wrong, for example on an ambiguous page. Kappa measures how often the 2 agree, so it cannot catch an error they share.
Key takeaways
- Report precision with an interval. 48 of 50 is somewhere between 87% and 99%, and that range is what a go-live decision rests on.
- Use the Wilson interval for proportions. The textbook interval claims certainty from 50 of 50 and goes above 100% near the top.
- Sample by stratum with a fixed hash. In my test, reruns reused 47 of 50 labels, against 1 with a plain random sample, and every segment is in the sample.
- Check a model labeler against people with Cohen's kappa. Raw agreement flatters a labeler when one answer is common; kappa corrects for chance.
- Gate on the lower bound, and against what is live. A small sample then fails by itself, and new data that is significantly worse than the current catalog does not ship.
- Keep the harness measurement-only. It writes a report, and a person makes the go-live call.
For how the same statistics drive the rollout of an LLM judge, see LLM as a judge in production: ship it in shadow mode first.
Footnotes
-
The Wilson interval inverts a score test: it is the set of true proportions p for which the observed count would not be rejected at the 5% level. Its center is pulled slightly toward 50%, which is why 50 of 50 gives a lower bound of 92.9%. For very small n, an exact Clopper-Pearson interval is more conservative still. I use Wilson because it is close to its stated coverage at the sample sizes I work with, and because it has a closed form. ↩
Common questions
What is Cohen's kappa?↓
It measures how much 2 raters agree beyond the agreement you would expect by chance. It is the observed agreement minus the chance agreement, divided by 1 minus the chance agreement. A kappa of 0 means the raters agree no more than chance would predict, and 1 means perfect agreement. It is useful when one label dominates, because raw agreement looks high even for a rater that always gives the common answer.
What is a Wilson score interval?↓
It is a confidence interval for a proportion, such as precision measured on a sample. It stays inside 0 to 1 and keeps close to its stated coverage at small sample sizes and near 0% or 100%, where the textbook interval breaks down. For 50 correct out of 50, the textbook interval is 100% to 100%, while the Wilson interval is about 93% to 100%.
How many rows do you need to label to evaluate LLM extraction?↓
Enough that the lower bound of the interval can clear your bar. If the bar is a 90% lower bound and the true precision is about 96%, 50 labeled rows gives a lower bound near 87% and fails, while 100 rows gives about 90% and passes. Work out the sample size from the bar before you label, and below 20 rows report that there is not enough data.
Can an LLM label data for evaluation?↓
Yes, if it is a different model from the one that produced the data, it judges against the source page, and you check it against human labels. I hand-label 60 rows and report percent agreement and Cohen's kappa per field. If kappa is low on a field, that field's model-labeled precision is not evidence.
Why stratify the evaluation sample?↓
Error rates differ by segment. Rows from one region, or from one website that supplies a quarter of the data, can fail in their own way. Stratifying makes sure each segment is in the sample, and weighting each stratum by its share of the population keeps the overall estimate unbiased.
What is considered a good Cohen's kappa score?↓
A widely used convention, from Landis and Koch in 1977, reads 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial and above 0.80 as almost perfect agreement. The bands are a convention, so report kappa per field: one weak field can hide inside a good average. On a field well below the substantial band, I treat model-labeled precision as unverified.
What is the difference between Cohen's kappa and weighted kappa?↓
Cohen's kappa counts every disagreement as equally wrong. Weighted kappa gives partial credit when the labels are ordered, so rating a 4 as a 3 counts as a smaller disagreement than rating it a 1. For yes-or-no labels, such as whether a row is in scope, the 2 give the same result.
Do you always use 1.96 for a 95% confidence interval?↓
1.96 is the z-value for a two-sided 95% interval under the normal distribution, and the Wilson interval uses it as well. For a one-sided bound, such as 'precision is at least X', the value is 1.645. Exact methods such as Clopper-Pearson avoid the normal approximation altogether.