Benchmarking on-device OCR

Capturing documents with a phone camera comes with a build decision: how do you turn the photo into structured fields your system can trust? A contact record, a receipt total, an ID number. The options in 2026 form three lanes:

flowchart TD
  P["Photo on a phone"] --> A["Lane 1: on-device OCR engine
  Apple Vision, ML Kit, PaddleOCR
  free, instant, offline"]
  P --> B["Lane 2: cloud vision model
  Gemini, GPT-class
  strongest reading, needs network"]
  P --> C["Lane 3: on-device LLM
  FoundationModels, Gemini Nano
  young, uneven device support"]
  A --> F["Structured fields"]
  B --> F
  C --> F

I hit this decision building a mobile capture feature, went looking for a measurement that answers it for phone photos, and found none. So I built one. This article is the method, the numbers, and a way to choose for your own constraints. My earlier OCR primer covers the fundamentals of why reading text from photos is harder than it looks; this one is about choosing and measuring.

The business version of the question is worth stating first, because it is what makes the accuracy numbers matter:

• A wrong extracted value is a downstream cost. A rep emails the wrong address, a reimbursement pays the wrong total, an audit fails. Extraction accuracy is data quality at the point of entry.
• Latency decides whether capture happens at all. A scan that confirms in half a second gets used at the moment of intent; one that spins for five seconds on venue wifi gets skipped, and the data never enters the funnel.
• The photo itself can be regulated data. IDs, medical forms, and badges with personal data carry residency and privacy constraints; whether the image leaves the device is a compliance property, decided by which lane you pick.

The public benchmarks cannot answer this

The obvious move is to check the leaderboards. Three reasons that fails for mobile capture:

1. They score the wrong inputs. OmniDocBench, the leading document benchmark, evaluates PDF pages and scans. OCRBench v2 and CC-OCR evaluate large multimodal models only. A phone photo brings perspective, glare, compression, and camera choice, and an assistive-tech study that measured OCR under real camera conditions found condition effects (distance, angle, lens) as large as engine effects.
2. They score the wrong output. Leaderboard metrics are text accuracy: character error rate, edit distance. Your system consumes fields. The two diverge sharply, which the next section quantifies.
3. The headline one is saturated. Top models now cluster above 94% on OmniDocBench, and its own maintainer ecosystem argues the exact-match scoring penalizes semantically correct output. A benchmark where everyone scores 94% ranks nobody.

And the engines developers ship on phones, Apple Vision and ML Kit, appear in none of the serious comparisons. The best-ranking web result comparing them is anecdotal and predates the current APIs by three years.

Score the fields you extract

Character error rate (CER) is the standard OCR metric, and for structured extraction it understates failure in a specific, predictable way: errors compound. A field is only correct if every character in it is correct, so per-character error behaves like compound interest running against you. At 2% CER, a 10-character field has about an 82% chance of coming out clean (0.98^10), and a document with five such fields extracts fully correct about 37% of the time. Published estimates for real receipts put the compounding cost at 15 to 20% field-extraction failure from a 2% CER, which matches what I measured.

So the benchmark scores per field. Each image has a ground truth like {name, title, company, email}, and output is compared field by field:

Metric	Question it answers	The failure it catches
Precision	Of the fields filled in, how many were right?	Wrong values a person then trusts
Recall	Of the fields that should be filled, how many were?	Gaps, visible and cheap to fix by hand
F1	Harmonic mean of the two	Lopsided engines hiding behind one good number

A worked example: the truth is {name, title, company}. The output has a correct name, a wrong title, and a blank company. Precision is 1 of 2 filled (50%), recall is 1 of 3 required (33%), F1 lands near 40%. That severity is correct for a record you intend to trust, because the two failure directions cost differently:

Recall failure: a blank field

• Visible at a glance
• Fixed on the spot by typing it in
• Costs seconds

Precision failure: a wrong value

• Looks exactly like a correct field
• Gets trusted and acted on
• Discovered weeks later, by the person it embarrassed

Values match in tiers: exact, then normalized (case, whitespace, punctuation), then fuzzy with a Levenshtein similarity floor of 0.85. Anna-Lena Weiß against Anna Lena Weiss counts; [email protected] against [email protected] does not.¹

The corpus: generated, seeded, and rough on purpose

There is no public dataset of badges, business cards, and profile screenshots with field-level ground truth, for a boring reason: real ones are personal data. You cannot scrape a thousand of them, and you could not publish the ground truth if you did. The same applies to most structured-extraction domains, which is why the method here transfers even if your documents are receipts or forms.

The corpus design:

• Rendered, then degraded. Documents rendered in a real browser from templates with known field values, then pushed through difficulty tiers, from clean through mild and field to hostile (rotation, glare, blur, torn edges).
• Seeded. Everything derives from one random seed, so the corpus regenerates byte-identical and every engine sees the same images. The engine head-to-head runs on 778 images stratified across type and tier; the full cloud-lane run uses 3,800.
• A negative corpus. Images with no document in them: signage, empty rooms. The correct output is nothing, and a lane passes only if it says so. This is where vision LLMs earn or lose trust.
• Harder than reality, on purpose. The hostile tier is a stress test. The clean and mild tiers represent the realistic majority; the point of the hostile tier is to find where each engine breaks, not to predict averages.

One more property: everything is scored end to end, through the same parsers and the same pipeline. A benchmark win the downstream parsers cannot use does not count.

Lane 1: on-device OCR engines

Engine, through a real mobile pipeline	Overall F1	badge	card	profile screenshot	clean	hostile	runs in React Native
Apple Vision (iOS)	85.4%	85.0	90.5	75.4	94.1	64.8	yes
ML Kit (Android)	75.5%	82.9	82.2	55.1	85.2	75.9	yes
PaddleOCR PP-OCRv4	74.7%	71.6	85.5	55.6	82.3	52.4	via native bridge
tesseract.js	59.0%	-	-	-	-	-	no

What the table says, in decision terms:

• On-device OCR’s reputation lags its quality. Apple Vision reads 94.1% of fields correctly on clean captures, instantly, free, offline, with zero empty recognitions across 778 images. It also degraded more gracefully on the hostile tier than the noise model I had assumed (64.8 measured against 46.5 assumed). If your mental model of on-device OCR is Tesseract-era, it is three generations stale.
• Check the runtime before the accuracy. tesseract.js scored 26 points behind Vision, and the score did not end up mattering: its WASM build cannot run inside React Native’s Hermes runtime at all. A laptop benchmark can miss the constraint that decides the question, so benchmark on the runtime you ship.
• The Android gap is real, specific, and not fixable by switching engines. ML Kit trails Vision by 9.9 points, concentrated in profile screenshots (55.1 vs 75.4) and cards (82.2 vs 90.5). PaddleOCR is statistically tied with ML Kit (+0.8, within sampling noise), and there is no Android system engine in Vision’s class. The fix is architectural (the hybrid section below), and the platform gap should feed your cost model: Android traffic will escalate to the cloud 2 to 3 times as often.²

Two follow-on results from operating this benchmark:

• Deterministic fixes beat prompt fixes when the failure is parsing. The weakest slice, profile screenshots, failed because interface chrome (follower counts, button labels) leaked into fields. Parser and heuristic tightening, no models involved, moved that slice from 75.8% to 90.9% F1 on real images, with title false positives falling from 70 to 6.
• A synthetic corpus can predict real failures and undersell real fixes. The same fix scored +6.8 points on synthetic images and +15.1 on real ones. Directionally right, conservatively wrong: the relationship you want from a synthetic benchmark.

A note on the 2026 API landscape, because it changes what this lane can do:

Capability	Apple Vision (iOS 26)	ML Kit v2	PaddleOCR
Text with boxes and confidence	yes	yes	yes
Tables, lists, document structure	yes, RecognizeDocumentsRequest	no	server-side variants
Detected data (emails, phones, URLs)	yes, built in	no, regex it yourself	no
Language coverage	26 languages	5 scripts	100+ (PP-OCRv5)

The big change here is iOS 26’s RecognizeDocumentsRequest: structured output (tables, lists, and recognized entities like emails and phone numbers) from the system framework, no LLM involved. For many extraction tasks that removes the reason people reach for a cloud model.

Lane 2: cloud vision models

Adapter	Overall F1	badge	hostile	$ per 1k photos	latency p50	hallucination on negatives
Gemini 2.5 Flash	90.5%	98.9	87.8	~$0.73	2.8s	0%
Gemini 2.5 Pro	87.4%	-	-	~$2.88	5.3s	110 truncated-JSON failures
Azure OpenAI, small models	78-80%	-	-	$0.14-0.41	-	13% of images blocked by content filter

The cost is worth deriving from first principles, because it generalizes: Gemini prices images by tiling them into 768x768 tiles at 258 tokens each. A full-resolution phone photo is about 20 tiles, roughly 5,200 input tokens, which at 2.5 Flash’s published $0.30 per million input tokens is about $0.0016 per photo before output tokens. My measured ~$0.73 per 1,000 scans agrees. At almost any business volume the per-unit cost is a rounding error; the real costs of this lane are latency, connectivity, and data egress.

Four findings that would have been guesses without measurement:

1. The bigger model lost. Gemini 2.5 Pro scored 3.1 points below Flash at 4x the cost and nearly double the latency, with 110 truncated-JSON failures on long outputs. For tight-schema extraction, the smaller model with better structured-output behavior won on every axis. Test the “use the biggest model” default; here it failed.
2. Hallucination direction is a model property. On my negative corpus, Flash returned a correct empty result on 100% of images. Published receipt benchmarks show the spread: some models fail conservative (missing fields, digit confusion), others fail aggressive, fabricating plausible invoice numbers to satisfy the schema. A fabricated value is worse than a missed one, because a missed scan gets retried and a fabricated one gets trusted. Mitigations that work: require the schema to permit emptiness, validate extracted values structurally (emails, checksums, arithmetic reconciliation on totals), and test negatives per model and prompt.
3. Content filters are part of the accuracy budget. On the Azure lane, 13% of images were rejected by the platform’s default content filter; a synthetic face next to contact text pattern-matches as content to block. The same corpus through Vertex: near zero. If your documents contain faces, measure the filter like you measure accuracy.
4. Vendor benchmark numbers do not transfer. Published OCR-vs-LLM comparisons put everyone at 95-98% on clean scans. On photos through a phone pipeline, the spread reopens. Measure on your own capture path.

If you want this lane without writing adapters, Zerox is the open source reference implementation (render each page, ask a cloud VLM for structured markdown). Same trade: strongest reading, cloud-only.

Lane 3: on-device LLMs

The young lane, and the one moving fastest:

• Apple FoundationModels (iOS 26, Apple Intelligence devices): a ~3B on-device model with guided generation. You declare a @Generable output struct and the runtime constrains decoding to the schema, so the output parses by construction. Paired with Vision’s OCR text, this is structured extraction with lane 1’s privacy and offline properties.
• Android, ML Kit GenAI on Gemini Nano: the Prompt API accepts image input, but structured output is Alpha and device support is a fragmented flagship list that differs by model generation. There is no Android equivalent of Apple’s uniform guarantee yet.

The platform asymmetry is the story of this lane in 2026: on iOS you can plan around it, on Android you cannot. I run it as an assist (filling only fields the OCR path left empty, so a model regression can never overwrite a value an engine read correctly) behind a flag, with the benchmark in place to score it when device coverage justifies more.

The hybrid: what the numbers support

No single lane wins the mobile-capture constraint set, and the measured architecture answer is a gate between two of them:

flowchart TD
  A["Photo captured"] --> K{Machine-readable code?}
  K -->|QR / barcode| Q["Decode on device, deterministic"]
  K -->|document photo| B["On-device OCR + parsers"]
  B --> R{Did clean fields parse?
  name-shaped name, valid email}
  R -->|yes, the majority| S["Accept instantly, offline"]
  R -->|no| C["Escalate to cloud VLM
  when connectivity allows"]
  C --> S
  Q --> S

Three design points carry it:

• Gate on output quality, not engine confidence. The escalation test is whether plausible fields parsed out of the text: a name shaped like a name, an email that validates. Weak reads identify themselves by their output, whichever engine produced them, and engine confidence scores stop being comparable across engines anyway.
• The gate converts a weaker engine into a usable one. Simulating this policy over the full corpus, Android’s effective accuracy rises from 75.5% on-device to about 88.9% blended, within 1 to 2 points of iOS, at the cost of escalating far more often. That trade (accuracy parity for cloud dependence) is exactly the number a product owner needs to accept or reject.
• Never send solved problems to a model. QR and barcode decoding is a deterministic algorithm with a right answer. Routing it through any model trades a solved problem for an unsolved one.

Choosing for your constraints

The reader-facing decision, condensed:

Your constraint	Pick
Must work offline, instant feedback	Lane 1, with lane 2 as background escalation
Photos are regulated data that cannot leave the device	Lane 1 (+ lane 3 on iOS for structure)
Server-side documents, no mobile capture	Lane 2 directly, with negative tests and value validation
Highest accuracy on degraded inputs, connectivity fine	Lane 2, small fast model first, bigger only if measured better
Structure (tables, entities) without any cloud	Apple Vision RecognizeDocumentsRequest on iOS 26
Web app	tesseract.js (v7) works there; it does not work in React Native

Key takeaways

• Score structured OCR per field. Character accuracy compounds against you: 2% CER can mean 15 to 20% field-extraction failure.
• Public leaderboards score PDFs and scans with text metrics, and the leading one is saturated. They cannot answer a phone-photo extraction question; a seeded corpus of your own documents can, at a few dollars of compute.
• On-device OCR is stronger than its reputation: 85.4% field-level F1 end to end, 94.1% clean, instant, free, offline.
• Benchmark on the runtime you ship. One engine lost 26 points and then turned out unable to run in the app at all.
• The bigger cloud model lost to the smaller one on accuracy, cost, latency, and output reliability. Measure before defaulting to size.
• Test hallucination with a negative corpus and check the failure direction: conservative models leave gaps, aggressive ones fabricate schema-satisfying values. Validate extracted values structurally either way.
• A parse-quality gate between on-device OCR and a cloud VLM gets within 1 to 2 points of the best single lane while keeping the offline path. Gate on whether fields parsed, never on engine confidence.
• Try deterministic fixes before prompt fixes: parser tightening moved the worst slice 15 points on real images, with no model in the loop.

Footnotes

1. There is no single right value for the 0.85 floor. I picked it so a one-character email typo fails while diacritics and hyphenation survive. Calibrate it per field type; emails and numbers need stricter matching than names. ↩

2. One comparability note. Every engine scored the same 50-per-cell stratified draw from the seeded corpus, but the TypeScript lanes and the Python PaddleOCR lane used independent shuffles of that draw, so the PaddleOCR row is matched in distribution rather than image-identical. The Vision and ML Kit rows share the identical 778-image set, so that head-to-head is exact. The inter-engine gaps (85.4 vs 75.5 vs 74.7) are far larger than the few-point sampling variance. ↩