The OCR Step Isn't Optional, Even When You're Using an LLM

Vision-language models are good enough now that it's tempting to point one at a raw document image and skip the OCR step entirely — fewer moving parts, one model instead of a pipeline. It works right up until the documents stop being clean, and in production, they always do.

Two separate failure modes, not one

OCR and LLM extraction fail differently, and conflating them is where a lot of document pipelines go wrong. OCR failures are mechanical and visible: low resolution, poor contrast, unusual fonts, and busy backgrounds degrade text recognition in ways that are obvious once you look — missing characters, garbled words, dropped lines. LLM extraction failures are quieter and more dangerous: the model produces output that's well-formatted and confident-looking but doesn't actually match the source document's structure, because the failure is in interpretation, not recognition. A pipeline that only benchmarks end-to-end accuracy can't tell which of those two problems it actually has, which means it can't fix the right one.

Classification is the single point of failure nobody budgets for

Before extraction even starts, most pipelines run a classification step to decide which schema applies to a given document — and if that step gets it wrong, every field extracted afterward is being pulled against the wrong template. This breaks in predictable ways: a batch upload that concatenates an invoice, a receipt, and a cover letter into a single PDF confuses a classifier built around the assumption of one document type per file, and the whole downstream extraction runs on a false premise nobody catches until the output looks wrong in a spot check.

• Benchmark layout accuracy and text-extraction accuracy separately — they fail independently and get hidden by a single blended metric.
• Feed the model both the OCR text and the document image where the tooling supports it — the combination consistently outperforms either alone.
• Test on real production documents, including multi-document bundles and handwritten pages, not a clean synthetic set.
• Add a reconciliation step that checks extracted totals and fields against the source before anything downstream trusts the output.

None of this is an argument against using LLMs for document extraction — they're a real improvement over rules-based parsing for anything with layout variation. It's an argument for treating OCR, classification, and extraction as three separate stages with three separate failure modes to monitor, instead of one black box you only notice is broken when a customer complains about a wrong number on an invoice.