Invoices, receipts and forms all have the same problem: the data is in a picture, and you need it in a database. Template-based OCR breaks on every new vendor layout. A vision-language model reads the page and returns fields directly. The catch is that it can be confidently wrong, so the work is in the pipeline around it. This post covers the general pipeline for any language. For Arabic documents, see reading Arabic invoices with vision for setup and Arabic OCR vs vision LLMs for when classic OCR is the better tool.
What does the pipeline look like?
- Prepare each page: render PDFs to images, fix rotation, resize.
- Extract: one request per document with a field-by-field prompt, JSON only.
- Validate: parse, schema-check, arithmetic-check.
- Retry once with the error message if validation fails.
- Review: anything still failing goes to a person, with the image and the draft.
- Measure field accuracy on a labelled sample, and re-run it whenever you change model or prompt.
How do you prepare document images?
Render PDFs to images with PyMuPDF and keep the smallest text sharp without sending more pixels than needed:
# pip install pymupdf pillow
import base64, io
import pymupdf # older code imports it as fitz
from PIL import Image
def pdf_pages_as_data_urls(path, long_edge=1800):
urls = []
for page in pymupdf.open(path):
pix = page.get_pixmap(dpi=200)
img = Image.open(io.BytesIO(pix.tobytes("png"))).convert("RGB")
img.thumbnail((long_edge, long_edge))
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=90)
urls.append("data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode())
return urls
On BroadNet Inference, images go in the message as data: URLs with an accurate MIME type, and on pay-as-you-go they bill as input tokens; larger images use more. Measure tokens per page once with the usage block and size your batches from that; counting tokens before sending shows how, and vision LLM cost per image turns tokens into cost per page. To estimate tokens per page before you send anything, drop a sample scan into the image token calculator.
How should you write the extraction prompt?
Be exact about keys, types, formats and missing values. If you also want a JSON Schema for validation or for providers that enforce one, the free JSON schema generator builds it from a sample invoice:
PROMPT = """Extract this invoice. Reply with one JSON object only, no prose.
Keys:
- vendor_name: string, as printed
- vendor_tax_id: string or null
- invoice_number: string
- invoice_date: string, YYYY-MM-DD
- currency: ISO 4217 code, e.g. USD, EUR, SAR
- line_items: list of {description: string, quantity: number, unit_price: number, line_total: number}
- subtotal: number
- tax: number or null
- total: number
Rules: numbers without currency symbols or thousands separators. If a field is not on the page, use null. Never guess."""
“Never guess” plus an explicit null matters most. Without it, models fill gaps with plausible values, which is worse than a blank because it passes a naive check.
How do you get structured output and validate it?
There are two levels of structured output. JSON mode guarantees valid JSON. Schema-constrained output also guarantees the shape. OpenAI’s guide puts it plainly: both produce valid JSON, but only Structured Outputs ensure schema adherence. Support for either varies by provider and model, so validate in code whatever the API promises:
import json, os
from decimal import Decimal
from pydantic import BaseModel, ValidationError, model_validator
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
class Line(BaseModel):
description: str
quantity: Decimal
unit_price: Decimal
line_total: Decimal
class Invoice(BaseModel):
vendor_name: str
vendor_tax_id: str | None = None
invoice_number: str
invoice_date: str
currency: str
line_items: list[Line]
subtotal: Decimal
tax: Decimal | None = None
total: Decimal
@model_validator(mode="after")
def sums(self):
tol = Decimal("0.02")
if abs(sum(l.line_total for l in self.line_items) - self.subtotal) > tol:
raise ValueError("line_totals do not sum to subtotal")
if abs(self.subtotal + (self.tax or 0) - self.total) > tol:
raise ValueError("subtotal + tax != total")
return self
def extract(image_urls, model="broadnet/qwen3.8-27b"):
content = [{"type": "text", "text": PROMPT}] + [
{"type": "image_url", "image_url": {"url": u}} for u in image_urls]
messages = [{"role": "user", "content": content}]
for attempt in range(2):
r = client.chat.completions.create(model=model, messages=messages, max_tokens=4000, temperature=0)
raw = (r.choices[0].message.content or "").strip().removeprefix("```json").removesuffix("```")
try:
return Invoice.model_validate(json.loads(raw)), r.usage
except (json.JSONDecodeError, ValidationError, ValueError) as e:
messages += [{"role": "assistant", "content": raw},
{"role": "user", "content": f"That failed validation: {e}. Re-read the image and reply with corrected JSON only."}]
return None, r.usage # send to human review
Use Decimal, not float, for money. Keep max_tokens generous: Qwen3.8 27B reasons before answering, and reasoning counts toward the limit; a small cap returns an empty string. See why reasoning models return empty replies.
Which checks catch the most errors?
| Check | Catches | Cost |
|---|---|---|
| JSON parses | Prose around the JSON, truncation | Free |
| Schema validates | Missing keys, wrong types | Free |
| Lines sum to subtotal | Misread digits, skipped lines | Free |
| Subtotal + tax = total | Misread total or tax | Free |
| Date parses and is not in the future | Day/month swaps, OCR noise | Free |
| Vendor tax ID matches your supplier list | Wrong vendor, fraud | One lookup |
| Duplicate invoice number per vendor | Double payment | One query |
Arithmetic checks are the best value: a single misread digit almost always breaks a sum. Receipts often lack line-level detail, so check what is present and accept fewer checks for them.
How do you measure accuracy before going live?
Label 50 to 100 real documents by hand, run the pipeline, and score per field, not per document. Totals and dates deserve a stricter bar than descriptions. Track three numbers: field accuracy, the share of documents that pass validation without retry, and the share sent to review. How to design that review step, including random audits of documents that passed, is in making human review part of your document AI workflow. When you try another model, run the same set; evaluating an LLM before switching covers the method.
Which vision models fit on BroadNet Inference?
| Model | Images | Context | Notes |
|---|---|---|---|
| Qwen3.8 27B | Yes | 262,144 | Tools, structured output and reasoning listed; tested in Arabic by BroadNet |
| Gemma 4 VL 26B | Yes | 32,768 | Lowest BroadNet price; runs on the free credit and the hourly key |
| DeepSeek V4.1 Flash | Yes | 1,048,576 | Images included on the unlimited plans |
Prices per 1M tokens are on the pricing page; multiply your measured tokens per page by monthly volume in the LLM API cost calculator.
Where BroadNet Inference fits
BroadNet Inference takes images on three chat models through the standard OpenAI message format, and it does not retain prompts, images or answers after the request, which matters for invoices full of names, tax IDs and bank details (zero data retention). Its docs list structured output for Qwen3.8 27B but do not document schema parameters, so rely on the validation layer above rather than on the API to enforce your schema. BroadNet does not publish an image size limit or a dedicated document-AI product, and its docs describe image input, not PDF upload, so render pages to images first. For high-volume runs on a fixed budget, an unlimited DeepSeek V4.1 Flash plan includes images.

