How to Validate LLM-Extracted Financial Data Before You Book It

You ran 40 bank statements through a model and got back a clean-looking spreadsheet in four minutes. Every row has a date, a description, an amount, and a balance. Nothing is blank, nothing looks broken, and that is exactly the problem: you have no idea which of those 3,000 rows are wrong.

Short Answer

Do not trust a model's confidence score to tell you where it failed. Validate in tiers instead: run arithmetic checksums first (running balance, opening plus movement equals closing, page totals), then cross-field logic rules (date ranges, sign conventions, decimal parsing), then a second independent read to surface disagreements, and only then send a targeted sample to a human. Tiers one and two are cheap, deterministic, and catch most of what actually goes wrong. Human review is the last tier because it is the expensive one.

The Errors LLMs Make Most Confidently

The failure mode that matters is not a refusal or a blank cell. It is a plausible wrong value with nothing attached to flag it. On long financial tables, the recurring patterns are:

Silent truncation. The model reads part of a multi-page table and stops, returning a well-formed result that is missing rows.
Value propagation. It misreads one date or reference number, then repeats that value down the column instead of re-reading each row.
Digit and sign errors. 1,234.56 becomes 123.456 under a different separator convention. A debit gets a positive sign. A parenthesised negative loses its parentheses.
Description drift. Merchant names get normalised or completed from the model's prior knowledge rather than transcribed.
Page-boundary loss. Rows that straddle a page break get dropped or merged.

None of these announce themselves. All of them are caught by arithmetic that the statement already contains.

The Four-Tier Validation Stack

Tier	What it is	Catches	Misses	Relative cost
1. Checksum math	Running balance, opening plus movement equals closing, printed page totals, transaction counts	Dropped rows, duplicated rows, digit errors, sign flips, truncation	Errors in non-numeric fields, offsetting errors that cancel out	Near zero, runs in code
2. Cross-field logic	Date-in-period, sort order, decimal precision, currency, account consistency, duplicate detection	Date misreads, locale parsing errors, page-merge artefacts, wrong account attributed	Semantically wrong but structurally valid values	Near zero, runs in code
3. Second read and disagreement diff	Extract twice with a different model, prompt, or render resolution, then diff cell by cell	Description drift, ambiguous glyphs, low-quality scan regions, anything one model guessed at	Errors both reads make identically, which is common on systematically bad scans	Doubles extraction spend, still cheap relative to labour
4. Targeted human review	Review flagged cells plus a stratified sample of unflagged rows	Almost everything remaining, if you route the right rows	Errors in rows you never sampled	The expensive tier, so spend it last

Have a messy PDF? Upload 1-3 sample pages and we will tell you if it is clean, OCR-heavy, or needs human QA.

Tier 1: Make the Document Check Itself

Financial documents are unusually good validation targets because they are internally redundant. Use that.

For bank statements, the strongest single check is the running balance chain. For every row, prior balance plus the signed amount should equal the stated balance. Walk the chain from the opening balance to the closing balance. A break at row 212 tells you the error is at or before row 212, which turns a 3,000-row review into a single-cell lookup. This one check catches dropped rows, duplicated rows, sign flips, and most digit misreads at once.

Layer these on top:

Opening balance plus total credits minus total debits equals closing balance, checked per page and per statement.
Extracted per-page subtotals match the subtotals printed on the page.
Transaction count matches the count printed on the statement, when one is printed.
For invoices and remittances, line items sum to the subtotal, tax is a consistent percentage of the taxable base, and subtotal plus tax equals the total.

Log which check failed and which row range it points at, then pass that range straight to tier four. One caveat: offsetting errors survive checksums. If the model swaps two amounts with identical balance impact, arithmetic alone will not see it. That is what the later tiers are for.

Tier 2: Cross-Field Logic Rules

These are assertions you write once and run on every job. They cost nothing to execute and they catch the class of error that checksums structurally cannot.

Date in period. Every transaction date falls inside the statement period on the header. Catches year misreads, common on statements printing two-digit years.
Sort order. Dates are non-decreasing when the source is chronological. A single out-of-order date usually means a page-merge artefact.
Decimal discipline. Every amount has exactly two decimal places and parses under one explicit locale. Do not let a parser guess between 1.234,56 and 1,234.56. Fix the locale from the header and assert it.
Sign convention. Debits negative, credits positive, consistently across all pages. Mixed conventions usually means the model re-inferred the schema mid-document.
Account consistency. The account number or last four digits is identical on every page assigned to one statement. Catches statements interleaved during a bulk upload.
Empty descriptions. A blank or one-character description is almost always a row the model half-read.
Exact duplicate rows. Same date, amount, description, and balance twice is worth flagging, even though genuine duplicates exist.

Tier 3: Two Reads Beat One Confidence Score

Model-reported confidence is the weakest signal in this stack. Ask a model how sure it is and you get a number reflecting fluency, not correctness, and invented values frequently score high. Token log-probabilities are better because they are not self-assessed, but they still measure how expected a token was, not whether it matches the pixels on the page. Treat either as a rough ranking for review order, never as a pass or fail gate.

Disagreement is the stronger signal. Extract the same document twice: different model, different prompt phrasing, or the same model against a page rendered at higher resolution. Diff the two outputs cell by cell. Cells where both reads agree are very likely correct. Cells where they diverge are your review queue, and that queue is typically a small fraction of the document.

If your extraction returns page citations or bounding boxes, use them. The value of a citation is not the model's claim that a number came from page 4. It is that you can crop that region and look at it without hunting through the PDF.

Calibrate any threshold against your own documents. Label a couple of hundred rows from your actual statement mix by hand, once, and measure what fraction of errors your rules catch. A threshold tuned on someone else's document set tells you nothing about yours.

Tier 4: Spend Human Attention Where It Pays

By the time you reach human review, most of the document should already be cleared. Route reviewers to:

Every row flagged by tiers one through three.
The first and last row of every page, where truncation and merge errors concentrate.
The largest absolute amounts, ranked. A wrong 12,400 costs more than a wrong 4.20.
A random stratified sample of the rows nothing flagged, so you can estimate your residual error rate rather than assume it is zero.

That last point is the one people skip. Without a clean random sample you have no error rate, only a feeling. A fixed sample of unflagged rows per job gives you a number you can report and track over time.

When a Different Approach Is Simply Better

Validation is worth building when PDFs are genuinely your only source. Often they are not, and we would rather say so:

Check for a structured export first. Most banks offer CSV, OFX, or QBO downloads covering the same period as the PDF. That data needs no validation stack because it was never guessed at.
Bank feed aggregators are the right tool for ongoing transaction data from live accounts, and the wrong tool for a closed account or a statement from 2013.
A configured IDP product makes sense if you process the same three layouts every month at volume. Templates are cheap to maintain when layouts are stable, and stable layouts are where template-based tools beat general models.

DataConvertPro sits in the case those options do not cover: mixed formats, scanned or historical documents, one-off backlogs, and statements from banks nobody has templated. We run the checksum and logic tiers on every job, put a human on the rows that fail, and hand back Excel or CSV. If a bank export or a template tool would solve your problem faster, we will tell you that instead.

Frequently Asked Questions

Can I just ask the model to double-check its own work?

A second pass by the same model on the same input tends to confirm the first answer rather than correct it. Self-verification helps most when you change something between passes: a different model, a higher-resolution render, or a prompt that asks the model to verify one specific claim against one specific page rather than review the whole table.

What accuracy should I expect before validation?

Anyone quoting a single accuracy number without seeing your documents is guessing. Clean digital PDFs behave very differently from a low-resolution scan of a fax. The useful question is your residual error rate after validation, measured on a sample of your own files. Build the measurement first.

Do checksums work on documents other than bank statements?

Yes, wherever the document contains its own arithmetic. Invoices have line items summing to totals and tax as a percentage. Payroll registers have gross minus deductions equals net. Brokerage statements have positions summing to account value. Ledger exports have debits equal to credits. Documents without internal arithmetic, such as contracts, rely more heavily on tiers three and four.

How much of the document should end up in human review?

That depends on scan quality and layout consistency, so treat any fixed percentage as a target rather than a promise. Size it by running tiers one through three on a pilot batch, counting what gets flagged, and pricing from there. If the flagged share is large enough that a human is effectively re-keying the document, the honest answer is that manual entry is the cheaper path for those files.