AI Invoice Data Extraction: How LLM Pipelines Work

Accounts payable teams still spend a surprising amount of time retyping invoices. Every supplier uses a different layout, some send scans, some send PDFs with tables that span pages, and purchase orders arrive in yet another format. AI invoice data extraction uses multimodal language models to read these documents the way a person would, then returns structured data your accounting system can use. This guide explains how we design these pipelines, where they beat template-based OCR, how to validate the output so errors do not reach your ledger, and a worked example you can adapt.

Why template-based OCR struggles with invoices

Traditional OCR tools extract text well, but turning that text into fields usually depends on templates or rules per supplier. That works for a handful of regular suppliers and breaks down when:

• New suppliers arrive every month, each needing a new template
• A supplier changes its layout and the template silently stops matching
• Line-item tables run across pages or wrap descriptions onto several lines
• Documents are scanned at an angle, photographed or stamped
• Invoices and purchase orders arrive in several languages

The result is a large exception queue and staff correcting the same fields again and again.

How AI invoice data extraction works

A multimodal model can look at the page image and the text together. Instead of matching a template, it is asked to fill a defined schema: supplier name, invoice number, dates, tax IDs, totals and every line item. A production pipeline wraps that model call in several stages.

1. Ingest: collect documents from a mailbox, upload folder or scanner, and record where each came from.
2. Pre-process: detect whether a PDF has a usable text layer, convert pages to images where needed, and split multi-invoice files.
3. Extract: send the page images (and text, where available) to the model with a strict output schema.
4. Validate: check the output with ordinary code, such as totals, tax arithmetic, ID formats and matching against purchase orders.
5. Review: route low-confidence or failed records to a person, with the source document shown alongside the extracted fields.
6. Post: create the bill or invoice in your accounting system or ERP, and keep a link back to the original file.

Research comparing image-based and text-based approaches to invoice extraction is available, for example this arXiv benchmark of multimodal versus text-based parsing. In practice, we test both on a sample of the client's own documents before choosing.

Define the schema before choosing a model

The schema is the contract between the model and the rest of your systems. We define it first, with the finance team, and validate every response against it. Pydantic is a convenient way to do this in Python; its documentation covers models and validators in depth.

Worked example: an invoice schema with built-in checks

The illustrative code below defines an invoice schema with line items, validates the arithmetic, and checks the format of an Indian GSTIN. The model call itself is left as a placeholder because the same pattern works with any provider that supports structured output.

import re
from datetime import date
from decimal import Decimal
from pydantic import BaseModel, field_validator, model_validator

GSTIN_RE = re.compile(r"^\d{2}[A-Z]{5}\d{4}[A-Z][1-9A-Z]Z[0-9A-Z]$")

class LineItem(BaseModel):
    description: str
    quantity: Decimal
    unit_price: Decimal
    tax_rate_percent: Decimal
    line_total: Decimal

    @model_validator(mode="after")
    def check_line_total(self):
        expected = (self.quantity * self.unit_price).quantize(Decimal("0.01"))
        if abs(expected - self.line_total) > Decimal("0.05"):
            raise ValueError(f"Line total {self.line_total} does not match qty x price {expected}")
        return self

class Invoice(BaseModel):
    supplier_name: str
    supplier_gstin: str | None = None
    invoice_number: str
    invoice_date: date
    po_number: str | None = None
    currency: str
    subtotal: Decimal
    tax_total: Decimal
    grand_total: Decimal
    line_items: list[LineItem]

    @field_validator("supplier_gstin")
    @classmethod
    def check_gstin(cls, v):
        if v and not GSTIN_RE.match(v):
            raise ValueError("GSTIN format looks wrong")
        return v

    @model_validator(mode="after")
    def check_totals(self):
        if abs(sum(li.line_total for li in self.line_items) - self.subtotal) > Decimal("0.10"):
            raise ValueError("Line items do not add up to the subtotal")
        if abs(self.subtotal + self.tax_total - self.grand_total) > Decimal("0.10"):
            raise ValueError("Subtotal plus tax does not equal the grand total")
        return self

def extract_invoice(page_images: list[bytes]) -> Invoice | None:
    raw_json = call_model(                      # placeholder for your provider's API
        images=page_images,
        instructions="Extract the invoice into the given JSON schema. Use null when a field is absent. Do not guess.",
        schema=Invoice.model_json_schema(),
    )
    try:
        return Invoice.model_validate_json(raw_json)
    except ValueError as err:
        send_to_review_queue(page_images, raw_json, str(err))
        return None

Two design choices matter here. The instruction tells the model to return null rather than guess, and the validation runs in ordinary code, not in the model. A model can make an arithmetic mistake; a validator will not.

Matching against purchase orders

Extraction is only half the job. For most finance teams, the real value comes from matching invoices against purchase orders and goods receipts before payment.

• Two-way match: invoice against purchase order, checking supplier, items, quantities and prices
• Three-way match: adds the goods receipt, confirming that what was invoiced was actually received

Set tolerances with the finance team, for example a small price variance per line, and send anything outside them to review with the differences highlighted. Store the PO number the model extracted, but always confirm it exists in your system before trusting it.

Keeping a person in the loop

Even a well-tuned pipeline should not post every invoice automatically from day one. We usually start with every invoice reviewed, then relax review as confidence grows:

1. Review everything, measuring which fields are corrected most often.
2. Auto-approve invoices from regular suppliers that pass every validation and match a PO.
3. Keep review for new suppliers, high-value invoices and anything that fails a check.

The review screen should show the original page next to the extracted fields, so a reviewer can confirm a value in seconds.

Measuring accuracy on your own documents

Do not rely on anyone's headline accuracy figure, including ours. Build a test set of real invoices from your suppliers, label the correct values once, and measure field-level accuracy for each pipeline version. Track supplier name, invoice number, dates, totals, tax and line items separately, because errors in each have different costs. Re-run the test set whenever you change the model, prompt or pre-processing.

Common pitfalls we help teams avoid

• Treating the model as the source of truth. The model proposes values; your validation rules and your purchase order data decide whether they are accepted.
• Ignoring multi-page documents. Line-item tables that continue onto a second page are where many pipelines quietly lose rows. Always process every page and check that line totals add up.
• Skipping supplier master data. Matching the extracted supplier name and tax ID against your vendor list prevents duplicate vendors and catches invoices from unknown senders.
• Forgetting duplicates. The same invoice is often emailed twice, or sent by email and by post. Check supplier plus invoice number before creating a bill.
• No feedback loop. If reviewer corrections are not analysed, the same mistakes repeat. A short weekly review of the most-corrected fields usually shows what to change.
• Automating payment too early. Posting a bill and paying it are separate decisions. Keep approval rules for payment in your accounting system, whatever the extraction confidence.

Security and data handling

Invoices contain bank details, tax IDs and pricing. Before sending them to any AI service:

• Check where the provider processes and stores data, and whether it uses your data for training
• Use API access with business data terms, not consumer tools
• Restrict who can see the review queue and the original files
• Keep an audit trail of every change a reviewer makes

Where this fits in a wider automation programme

Invoice extraction is often the first step in removing manual entry across finance and sales operations. For suppliers that send consistent, machine-generated PDFs, a rules-based approach can be cheaper; our guide to parsing PDF and Excel invoices with custom Python scripts covers that route. The same ideas also apply to emails and contracts, as we explain in automating repetitive data entry into your CRM.

Frequently Asked Questions

How accurate is AI invoice data extraction?

It depends on your documents, the model and the validation around it. The only reliable answer comes from testing on a labelled sample of your own invoices. Validation rules and human review catch most errors before they reach your accounts.

Can it handle scanned and handwritten invoices?

Multimodal models can read scanned and photographed documents, and often cope with stamps and skewed pages. Handwriting is less reliable, so route those documents to review.

Do we need to replace our accounting software?

No. The pipeline produces structured data and posts it through your accounting system's or ERP's API, or prepares an import file if no API is available.

What happens when the model gets a field wrong?

Validation catches many errors, such as totals that do not add up or a malformed tax ID, and sends the invoice to review. Reviewer corrections are logged, which shows where the prompt or pre-processing needs work.

Explore AI document automation with us

We design extraction pipelines that read invoices and purchase orders, validate them and post them into your accounting, ERP or CRM systems. See our AI automation solutions, or talk to us about automating your invoice processing.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.