BroadRouter

Extract invoice, receipt and form data to JSON with a vision LLM

By the BroadNet Inference team · 4 min read · Published

Render each page to an image, send it to a vision model with a prompt that names every field and its format, and ask for JSON only. Then validate in code: parse it, check its schema, check that line items add up to the total, and retry once with the error. Route repeat failures to a person.

A paper invoice headed "فاتورة" (invoice) with tabs labelled Vendor, Date and Total, and the text "Turn documents into useful data."
Illustrative example.

Invoices, receipts and forms all have the same problem: the data is in a picture, and you need it in a database. Template-based OCR breaks on every new vendor layout. A vision-language model reads the page and returns fields directly. The catch is that it can be confidently wrong, so the work is in the pipeline around it. This post covers the general pipeline for any language. For Arabic documents, see reading Arabic invoices with vision for setup and Arabic OCR vs vision LLMs for when classic OCR is the better tool.

What does the pipeline look like?

  1. Prepare each page: render PDFs to images, fix rotation, resize.
  2. Extract: one request per document with a field-by-field prompt, JSON only.
  3. Validate: parse, schema-check, arithmetic-check.
  4. Retry once with the error message if validation fails.
  5. Review: anything still failing goes to a person, with the image and the draft.
  6. Measure field accuracy on a labelled sample, and re-run it whenever you change model or prompt.

How do you prepare document images?

Render PDFs to images with PyMuPDF and keep the smallest text sharp without sending more pixels than needed:

# pip install pymupdf pillow
import base64, io
import pymupdf  # older code imports it as fitz
from PIL import Image

def pdf_pages_as_data_urls(path, long_edge=1800):
    urls = []
    for page in pymupdf.open(path):
        pix = page.get_pixmap(dpi=200)
        img = Image.open(io.BytesIO(pix.tobytes("png"))).convert("RGB")
        img.thumbnail((long_edge, long_edge))
        buf = io.BytesIO()
        img.save(buf, format="JPEG", quality=90)
        urls.append("data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode())
    return urls

On BroadNet Inference, images go in the message as data: URLs with an accurate MIME type, and on pay-as-you-go they bill as input tokens; larger images use more. Measure tokens per page once with the usage block and size your batches from that; counting tokens before sending shows how, and vision LLM cost per image turns tokens into cost per page. To estimate tokens per page before you send anything, drop a sample scan into the image token calculator.

How should you write the extraction prompt?

Be exact about keys, types, formats and missing values. If you also want a JSON Schema for validation or for providers that enforce one, the free JSON schema generator builds it from a sample invoice:

PROMPT = """Extract this invoice. Reply with one JSON object only, no prose.
Keys:
- vendor_name: string, as printed
- vendor_tax_id: string or null
- invoice_number: string
- invoice_date: string, YYYY-MM-DD
- currency: ISO 4217 code, e.g. USD, EUR, SAR
- line_items: list of {description: string, quantity: number, unit_price: number, line_total: number}
- subtotal: number
- tax: number or null
- total: number
Rules: numbers without currency symbols or thousands separators. If a field is not on the page, use null. Never guess."""

“Never guess” plus an explicit null matters most. Without it, models fill gaps with plausible values, which is worse than a blank because it passes a naive check.

How do you get structured output and validate it?

There are two levels of structured output. JSON mode guarantees valid JSON. Schema-constrained output also guarantees the shape. OpenAI’s guide puts it plainly: both produce valid JSON, but only Structured Outputs ensure schema adherence. Support for either varies by provider and model, so validate in code whatever the API promises:

import json, os
from decimal import Decimal
from pydantic import BaseModel, ValidationError, model_validator
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

class Line(BaseModel):
    description: str
    quantity: Decimal
    unit_price: Decimal
    line_total: Decimal

class Invoice(BaseModel):
    vendor_name: str
    vendor_tax_id: str | None = None
    invoice_number: str
    invoice_date: str
    currency: str
    line_items: list[Line]
    subtotal: Decimal
    tax: Decimal | None = None
    total: Decimal

    @model_validator(mode="after")
    def sums(self):
        tol = Decimal("0.02")
        if abs(sum(l.line_total for l in self.line_items) - self.subtotal) > tol:
            raise ValueError("line_totals do not sum to subtotal")
        if abs(self.subtotal + (self.tax or 0) - self.total) > tol:
            raise ValueError("subtotal + tax != total")
        return self

def extract(image_urls, model="broadnet/qwen3.8-27b"):
    content = [{"type": "text", "text": PROMPT}] + [
        {"type": "image_url", "image_url": {"url": u}} for u in image_urls]
    messages = [{"role": "user", "content": content}]
    for attempt in range(2):
        r = client.chat.completions.create(model=model, messages=messages, max_tokens=4000, temperature=0)
        raw = (r.choices[0].message.content or "").strip().removeprefix("```json").removesuffix("```")
        try:
            return Invoice.model_validate(json.loads(raw)), r.usage
        except (json.JSONDecodeError, ValidationError, ValueError) as e:
            messages += [{"role": "assistant", "content": raw},
                         {"role": "user", "content": f"That failed validation: {e}. Re-read the image and reply with corrected JSON only."}]
    return None, r.usage  # send to human review

Use Decimal, not float, for money. Keep max_tokens generous: Qwen3.8 27B reasons before answering, and reasoning counts toward the limit; a small cap returns an empty string. See why reasoning models return empty replies.

Which checks catch the most errors?

Check Catches Cost
JSON parses Prose around the JSON, truncation Free
Schema validates Missing keys, wrong types Free
Lines sum to subtotal Misread digits, skipped lines Free
Subtotal + tax = total Misread total or tax Free
Date parses and is not in the future Day/month swaps, OCR noise Free
Vendor tax ID matches your supplier list Wrong vendor, fraud One lookup
Duplicate invoice number per vendor Double payment One query

Arithmetic checks are the best value: a single misread digit almost always breaks a sum. Receipts often lack line-level detail, so check what is present and accept fewer checks for them.

How do you measure accuracy before going live?

Label 50 to 100 real documents by hand, run the pipeline, and score per field, not per document. Totals and dates deserve a stricter bar than descriptions. Track three numbers: field accuracy, the share of documents that pass validation without retry, and the share sent to review. How to design that review step, including random audits of documents that passed, is in making human review part of your document AI workflow. When you try another model, run the same set; evaluating an LLM before switching covers the method.

Which vision models fit on BroadNet Inference?

Model Images Context Notes
Qwen3.8 27B Yes 262,144 Tools, structured output and reasoning listed; tested in Arabic by BroadNet
Gemma 4 VL 26B Yes 32,768 Lowest BroadNet price; runs on the free credit and the hourly key
DeepSeek V4.1 Flash Yes 1,048,576 Images included on the unlimited plans

Prices per 1M tokens are on the pricing page; multiply your measured tokens per page by monthly volume in the LLM API cost calculator.

Where BroadNet Inference fits

BroadNet Inference takes images on three chat models through the standard OpenAI message format, and it does not retain prompts, images or answers after the request, which matters for invoices full of names, tax IDs and bank details (zero data retention). Its docs list structured output for Qwen3.8 27B but do not document schema parameters, so rely on the validation layer above rather than on the API to enforce your schema. BroadNet does not publish an image size limit or a dedicated document-AI product, and its docs describe image input, not PDF upload, so render pages to images first. For high-volume runs on a fixed budget, an unlimited DeepSeek V4.1 Flash plan includes images.

Questions

Can a vision LLM replace OCR for invoices?

For many layouts, yes. A vision model reads the image and maps text to fields in one step, without templates per vendor. It still misreads digits sometimes, so keep validation and a review queue.

How do I get reliable JSON from a vision model?

List the exact keys, types and formats in the prompt, ask for JSON only, and parse it in code. If the model supports schema-constrained output, use it. Either way, validate with a schema library and retry once with the error.

How do I extract data from a multi-page PDF invoice?

Render each page to an image, then send all pages of one invoice in a single request if they fit the model's context, so totals on the last page can be matched to lines on the first.

What resolution should invoice images be?

High enough that the smallest digits are sharp, and no higher. Larger images use more input tokens. A long edge of about 1,500 to 2,000 pixels is a sensible starting point for an A4 page; test on your own documents.

How do I measure extraction accuracy?

Label 50 to 100 real documents by hand and score field by field. Track totals and dates separately from free-text fields like vendor names, because their error costs differ.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity