Tutorial 3 · End-to-end · Finance / Operations

Invoice field extraction with the span-extraction recipe

By the end of this tutorial you'll have a small language model that takes free-form invoice text and emits a JSON object naming the vendor, total, line items, invoice date, and due date — with character offsets for every span so your downstream system can audit exactly which tokens produced each value. It runs on a single small GPU (or CPU), survives layout changes that would break a regex, and deploys as an inline microservice on every invoice ingest.

Level: intermediate Time: ~2.5 hours total (most of it answer-key span tagging, which is the work that matters) Prerequisites: Tutorial 0 (Setup BrewSLM). Optional context: Tutorial 2 for per-class-floor framing, Task shapes for span-extraction vs classification.

Before you start

This tutorial assumes BrewSLM is running locally at http://localhost:5173 with an admin user signed in. If you haven't done that yet, complete Tutorial 0 — Set up BrewSLM and your first project first. It takes ~15 minutes and is the prerequisite for every tutorial in this track.

You'll also want, before you start: ~100 invoice texts (OCR'd from your AP scans, exported from your ERP, or sampled from FUNSD / CORD — see below) plus the corresponding ERP entries for vendor, total, and dates. The ERP entries are what bootstrap your gold spans so you're not annotating from scratch.

Terms you'll see in this tutorial (click to expand)
Recipe
The training-plan template behind a project. For this tutorial: span-extraction (shown in the app as the task type Structured Span Extractor / Extract structured fields). Defines the base model + adapter + pass/fail-rule defaults for structured-extraction tasks.
Answer key
BrewSLM's name for the gold set — the trusted rows every run is scored against. It lives under Data → Labels (the Answer key card). This tutorial says "answer key" for the thing in the app and "gold" for the idea.
Adapter
The mapping layer that converts your CSV/JSONL rows into training-ready fields. For span-extraction the adapter is default-canonical — it reads text plus a JSON list of {type, start, end, text} entity objects.
Task profile
The shape category the platform uses to pick eval handlers + synth playbooks. Span-extraction's task profile is structured_extraction — different from classification (one label per row) and from rag_qa (free-text answer).
Scoring mode
How the eval handler compares prediction to gold. Span-extraction uses span_set — the prediction and gold are each sets of (type, start, end) tuples, and the score is the F1 of set overlap. This is NOT the same as field_match (which would require an exact-string match on a single field).
Entity
One tagged span in a row. For invoices: vendor, total, line_item, invoice_date, due_date. Every entity is a tuple of (type, character-start-offset, character-end-offset, raw-text).
Span-set F1
The headline metric. Precision = correctly-predicted entities / total predicted; recall = correctly-predicted entities / total in gold; F1 = harmonic mean. An entity is "correct" only when type AND offsets both match.
Per-entity-type precision
The same precision metric, sliced per entity type. For invoice extraction you'll set a tighter precision floor on total (a wrong total causes a wrong payment) than on line_item (one missed line item is recoverable downstream).
Stratified-by-template split
Train/val/test split where each split contains different invoice templates (vendors, layouts). Prevents the model from memorising a specific layout and pretending it generalises.

This is BrewSLM's canonical workflow for structured extraction from free-form text — pulling named fields out of documents where the position, formatting, and surrounding context all vary. Invoice extraction is the headline use case but the recipe shape generalises: pulling named-entity offsets from contracts, statements, receipts, purchase orders, expense reports, ID documents. The model learns which tokens carry each field; you don't have to re-write your regex stack every time a vendor changes their template.

The end state is the small model you'd actually run in production at a B2B finance team: small enough to deploy as middleware on the ingest path, accurate enough that AP staff stop double-checking every extraction, and honest enough about its uncertainty that low-confidence rows route to a human reviewer instead of straight into the ERP.

What you'll build

A span-extraction model that takes invoice text and returns a JSON object with five entity types and their character offsets. Concretely, given an input like:

ACME Office Supplies Co
Invoice #INV-2026-00471
Bill Date: 2026-05-12   Due: 2026-06-11

Item                  Qty   Unit    Total
Premium copy paper     20   12.00   240.00
Stapler — heavy duty    3   18.00    54.00
Box of ballpoint pens   8    4.50    36.00

                                     ------
                            TOTAL:  $330.00

… the model emits:

{
  "entities": [
    {"type": "vendor",       "start": 0,   "end": 23,  "text": "ACME Office Supplies Co"},
    {"type": "invoice_date", "start": 67,  "end": 77,  "text": "2026-05-12"},
    {"type": "due_date",     "start": 84,  "end": 94,  "text": "2026-06-11"},
    {"type": "line_item",    "start": 140, "end": 178, "text": "Premium copy paper     20   12.00   240.00"},
    {"type": "line_item",    "start": 179, "end": 217, "text": "Stapler — heavy duty    3   18.00    54.00"},
    {"type": "line_item",    "start": 218, "end": 257, "text": "Box of ballpoint pens   8    4.50    36.00"},
    {"type": "total",        "start": 311, "end": 318, "text": "$330.00"}
  ]
}

The model is a fine-tuned LoRA adapter on top of SmolLM2-135M-Instruct running at typical 20-80ms per invoice on a single GPU. The output is a clean JSON object — your AP automation calls the microservice, gets back the structured fields, posts them straight to the ERP for any invoice it's confident about, and routes the rest to a human queue.

Key idea

A span-extraction model knows which tokens are the value, not just that a value exists. That's why character offsets are first-class — your downstream system can highlight the exact bytes that produced "$330.00" in the original document, audit trails get cheap, and a bad extraction is a fixable bad extraction rather than a black box.

Why not regex (and why not a frontier LLM API)

Three options exist for invoice field extraction. Use this comparison:

ApproachLayout robustnessLatencyCost at 10k invoices/dayPrivacy
Regex / rule-based extractorsBreaks every time a vendor changes template; rules accrete forever<5msFree (but a full-time eng to maintain)Self-hosted
Frontier LLM via APIExcellent — handles novel layouts1.5-4 seconds per invoice~$300-$2,000/month per 10k invoices/day, depending on token volumeEvery invoice (with PII + financial data) leaves your network
Small fine-tuned span extractor (this tutorial)Good across templates if you train on layout diversity20-80ms per invoice~$30/month (one GPU, amortised)Self-hosted, audit-ready

Regex was the right answer in 2015. It stopped being the right answer the day your sixth vendor sent an invoice that put the total in a top-right corner instead of a bottom-right one, and your seventh vendor decided that "Bill Date" meant something different from "Invoice Date". Every new template adds a rule; every rule rots when the vendor tweaks their template; the regex stack grows monotonically until nobody on the team can change it without breaking three things.

Frontier-LLM-via-API solves the layout problem but costs money per call, adds 1-4 seconds of latency to every ingest, and ships your customers' invoice data (vendor names, line items, totals — material non-public financial info) to a third-party. For most finance teams that's a regulatory non-starter even before the cost shows up on the bill.

Small fine-tuned models occupy the gap: layout-robust like an LLM, fast and self-hosted like a regex.

Why span-extraction, not classification or QA?

Within BrewSLM you have three obvious recipes you could point at this problem. Pick the one that matches the question you actually need answered:

You need WHICH tokens, not just IS this an invoice. That's the span-extraction signature.

Choose your dataset

You need invoice text paired with the entity spans you want extracted. Four common starting points; mix and match:

Your own AP system (the spine)
The highest-signal data you can get. Export 100-300 recent invoices as plain text (PDF → OCR if you have to) paired with the ERP entries that captured vendor / total / dates. Your gold spans bootstrap from those ERP entries — see the gold-set section. This is the work that actually matters; everything else is volume.
FUNSD — Form Understanding in Noisy Scanned Documents
FUNSD ships 199 fully-annotated scanned forms with per-token labels and bounding boxes. Small but high-quality — useful for validating the workflow and for diversifying your gold set with non-invoice form layouts.
CORD — Consolidated Receipt Dataset
CORD has 1,000 receipt images with structured-field annotations (menu items, totals, sub-totals, taxes). Closer to your invoice domain than FUNSD; use it to seed line-item examples.
DocBank — large-scale document layout
DocBank is 500K+ documents with token-level layout annotations. Overkill for an invoice extractor, but useful as a pre-training corpus if you decide to swap to a larger base model later.

Public data is for diversity, not coverage

Your AP system's invoices are what your model will see in production. FUNSD and CORD round out template diversity so the model doesn't memorise your top three vendors. Treat them as a 20-30% supplement to your own data — never the spine. A model trained 100% on CORD receipts will be brilliant at coffee shop receipts and useless at your actual vendor invoices.

Ingest and map

In BrewSLM, create a new project: Projects → + New Project, give it a name and describe the job in What problem do you want this model to solve? ("extract vendor, total, line items and dates from invoice text, with character offsets"). The project opens on its Overview. Then pin the task shape: Data → Labels → What kind of task is this? → Change → Extract structured fields (the span-extraction recipe; Train → More options → Task type → Structured Span Extractor → Use this task type is the same switch). Confirming it sets the adapter (default-canonical), task profile (structured_extraction) and the scaffolded pass/fail rule set (span-set F1 + precision + recall).

Open Data → Sources → Import dataset (auto-mapping) → Import dataset →. The wizard walks Source → Task type → Map → Preview & Confirm; point it at your JSONL. The canonical span-extraction shape is:

{
  "text": "ACME Office Supplies Co\nInvoice #INV-2026-00471\nBill Date: 2026-05-12   Due: 2026-06-11\n\nTOTAL: $330.00",
  "entities": [
    {"type": "vendor",       "start": 0,   "end": 23,  "text": "ACME Office Supplies Co"},
    {"type": "invoice_date", "start": 41,  "end": 51,  "text": "2026-05-12"},
    {"type": "due_date",     "start": 58,  "end": 68,  "text": "2026-06-11"},
    {"type": "total",        "start": 78,  "end": 85,  "text": "$330.00"}
  ],
  "rationale": "Standard layout; date label is 'Bill Date' rather than 'Invoice Date'."
}

Two required fields: text (the full invoice text) and entities (a JSON array of {type, start, end, text} objects). The optional rationale is a free-text note about what edge case this row exercises — useful for you as a reviewer, ignored by the trainer.

The wizard's Map step scans your file's columns and proposes the mapping. For JSONL with the canonical shape it usually picks the right fields automatically. Preview & Confirm shows the accepted sample rows (and the rejected ones, grouped by reason); click Import N row(s) to project when the preview looks right.

✓ Checkpoint: Data → Sources lists the import (Details → Sources summary shows the row count per source, e.g. "147 rows"). The app doesn't tally rows per entity type at import, so do it yourself with a one-liner over the JSONL — how many rows have at least one vendor span, how many have a total, etc. If any entity type has under 30 rows, you'll need more answer-key rows for that type before training; the Readiness overview ledger (same Details drawer) flags thin data on its data_ready row.

Character offsets must match the text exactly

The start and end values are byte (well, codepoint) offsets into the text field. If you copy-paste an invoice from a PDF into the answer-key form and the PDF inserted U+00A0 (non-breaking space) where you expected a regular space, your offsets will be off by some N and the eval handler will score every prediction wrong. The Data → Prepare step (next section) normalises whitespace; do it before you annotate, not after.

Cleanup: OCR artefacts and currency normalisation

Open Data → Prepare (chunk size, Redact PII & Secrets, Clean All Documents; one-click autofixes for the common cases) and its Details → Quality and safety section. For invoice text you're cleaning four common artefact families:

You don't have to clean everything before training. You do have to clean the rows you're about to put in the answer key — those are the rows the pass/fail rules score against, and their offsets need to be exact.

Pick the recipe: span-extraction or something else?

The decision tree for invoice extraction:

You want…UseWhy
Character offsets for every extracted fieldspan-extractionJSON list of {type, start, end, text}; downstream system can audit + highlight
A yes/no flag on whether a document is an invoice at allclassificationOne label per document; useful for ingest triage, not extraction
A free-text answer to "what's the total on this invoice?"qa-sft or rag-protocolConversational; not structured enough for an ERP integration
An executive summary of each invoicesummarizationDifferent shape entirely
Extract fields + flag fraud risk in one passspan-extraction first, then a small classifier downstreamTwo narrow models compose better than one wide one

For the canonical AP-automation use case: span-extraction. Sticking with it for the rest of this tutorial.

Domain packs (the finance gap)

BrewSLM doesn't ship a finance-domain pack out of the box today — the curated packs (support, ecommerce, legal, healthcare; Settings → Tools → Domain Packs) are around content domains, not finance/AP workflows. For invoice extraction you're operating on the generic cleaning recipe defaults plus the span-extraction recipe defaults, which is fine for a first project.

Building a custom finance pack is a worthwhile follow-up that this tutorial intentionally doesn't cover. It would bundle: stricter precision floors on total and due_date (the entities whose errors cause real-world payment failures), currency normalisation recipes for your operating regions, a glossary linking eval gates to SOX / regulatory-control language, and an Academy tag pointing at this tutorial. If your team is shipping multiple AP extractors (invoices, purchase orders, statements), packaging the conventions as a domain pack pays back fast.

Build the answer key (gold set) — manual spans + LLM-assisted promotion

The answer key — BrewSLM's name for the gold set — is where this tutorial diverges most from tutorial 2. Classification gold is a yes/no label per row; span-extraction gold is a list of typed spans per row, each with exact character offsets. Two complementary paths:

Path A — manual span tagging

Open Data → Labels. The Answer key card holds it (Practice answer key / Final answer key picker, Lock), and for this task shape its add-entry form is in span mode. For each row:

  1. Paste the invoice text into the text field — verbatim, already cleaned of OCR artefacts.
  2. For each field you want extracted, highlight the span in the text, type its entity type (vendor / total / line_item / invoice_date / due_date) and click + Add highlighted span. The form appends {type, start, end, text} to the entities JSON and writes the character offsets for you.
  3. Add a one-line rationale if the row covers a weird edge case ("Date label is 'Bill Date' not 'Invoice Date'", "Total is split across two lines"). The rationale isn't used for training but it's invaluable when a future you is debugging an eval failure six weeks from now.

Spend an hour here. Aim for 150 gold rows minimum — invoices vary more than classification inputs, and the model needs diversity across templates to generalise. Quality beats volume but coverage of edge cases beats quality on a single template.

Path B — LLM-assisted promotion from ERP records

This is the highest-leverage path if you have an existing ERP. Your AP system already knows the vendor, total, and dates for every paid invoice — those are the gold values for three of the five entity types. A short script plus the import wizard turns this into a 30-minute job:

  1. Bulk-import your invoice texts as raw rows.
  2. Join each row with its ERP entry — supply (vendor_name, total_amount, invoice_date, due_date) alongside the invoice text in your import file.
  3. Run a teacher model (Ollama / OpenAI / Anthropic) via a small script that feeds it each (invoice text, ERP values) pair and asks the teacher to locate each ERP value inside the invoice text and emit the corresponding span offsets in the platform's canonical entity-JSON shape. Use the platform's API key + a quick script — there's no built-in "seed-from-metadata" UI for this flow today; it's straight backend scripting against the synth backend you've configured. (Line items, which the ERP usually doesn't itemise, you still tag by hand.)
  4. Import the teacher's output through Data → Sources → Import dataset. The rows land as pending synthetic rows in the Review queue (Data → Generate → Details → Review queue). Accept the good ones — they become training rows; the ones you want to score against, add to the answer key on Data → Labels.

This trades 4-6 hours of manual offset-tagging for 30 minutes of review. The trade-off is the same as in tutorial 1: you have to actually review the rows. The platform will not auto-accept LLM-generated rows; every acceptance is an explicit decision.

✓ Checkpoint: the Answer key card on Data → Labels shows your row count, and the Readiness overview ledger (Data → Sources → Details) turns its gold_set row green once you're past the recipe's recommended count (amber below it, e.g. "60 rows · 150 recommended"). The amber state is fine for now — you'll grow it with synth in the next sections. Tally rows per entity type yourself: every entity type should have at least 30 examples in the answer key. If due_date is sitting at 8 rows because half your invoices have "Net 30" instead of an explicit due date, that's signal — you need more rows with explicit due dates AND a few rows with "Net 30" labeled correctly.

Don't skip the "no due date" examples

Your answer key must contain rows where some fields are legitimately absent — invoices with no due date, invoices with no itemised line items, statements with no single "total" line. The model needs to learn that absence is a valid output (an empty entity list for that type), not a hallucination cue. Without these rows in gold, the model will invent a due date for every invoice and your span-set precision tanks.

Stratified split by invoice template

Data → Split (the Train / Val / Test Split form) cuts the split when you click Run Split. For span-extraction on invoices, one override is non-negotiable:

For 250 rows across 12 vendors, an 80/10/10 disjoint split produces roughly 200 train / 25 val / 25 test with no template shared between splits. The 20-row new-vendor benchmark lives separately and is your tripwire for template generalisation. On every later re-split tick Keep the current N test examples so new data never moves the answer key into training.

Data → Split: the Train / Val / Test Split form with the split adapter, preset buttons and adapter config JSON; the ratio, seed, Stratify By Field and Disjoint By Field inputs follow below.
Data → Split. Disjoint By Field sits under the ratios; Run Split cuts the split and snapshots a prepared version.

Generate paraphrase + hard-negative drills

Three playbooks matter here (Data → Generate → Details → Playbooks). For invoice extraction the headline ones are paraphrase and hard-negatives:

Paraphrase positives (POSITIVES_PARAPHRASE) — coverage extender
Vary the surrounding text of a gold row while keeping every entity span verbatim. New header, different vendor address line, a re-ordered set of preamble fields, but "$330.00" still appears in the same form and gets re-tagged at its new offset. Goal: the model learns to find the entity regardless of what surrounds it. Generate ~80 rows seeded from your manually-curated gold.
Hard negatives (HARD_NEGATIVES) — the precision-defender drill
Look-alike strings that AREN'T the target entity. The most useful hard-negative class for invoices is dates in headers vs the invoice date: scanner output that includes a "downloaded on 2026-04-02" timestamp at the top of the page, OR an order date elsewhere in the document, where the model needs to learn that those are not the invoice_date. Same for totals — "subtotal", "tax", "shipping", "amount due", and "total" all look like total candidates. The hard-negatives playbook generates rows where these look-alikes are present but the correct entity is somewhere else. Generate ~50 rows.
Target a failure cluster (CLUSTER_TARGETED) — fill the under-performing slice
After a first round of eval surfaces failure clusters, this playbook seeds new examples from rows in the worst-performing cluster. Optional — only run after your first training pass produces an eval result with non-trivial failure clusters. The failure clusters on Results → Failures launch it directly (Generate N rows from this cluster).

Open Data → Generate. Set Generation Mode to PII / NER span extraction (the span mode — the name says PII, the entity types are yours: vendor, total, line_item, invoice_date, due_date), pick a Provider (Local (Ollama) is the free default) and Model Name, then open Details → Playbooks and run Paraphrase positives first with a target of 80. Generation runs as a background Job; the notification bell tracks progress and pings when it finishes. The span mode also carries the span tools (Refine weak types, Write answer key with <model>, labelling conventions, Add a type) — Tutorial 7 uses them in anger.

Run paraphrase before hard-negatives

Unlike the classification tutorial — where hard-negatives are the precision spine and run first — span-extraction starts with paraphrase because the model has to learn where entities live across layouts before it can learn what NOT to grab. Run paraphrase to ~80 rows, review them, train a quick baseline. Then run hard-negatives, review, retrain. Two cycles, not one giant generation pass.

Review the synth queue

Every generated row lands pending in the Review queue (Data → Generate → Details → Review queue). The queue groups rows by the source that produced them so you can review one category at a time; tick rows and use Accept selected / Reject selected (the Review queue under Data → Labels → Details has Accept all / Reject all per source group). Per-row actions:

Expect to reject 20-30% of paraphrase rows and 30-50% of hard-negative rows on first pass. Hard negatives are harder for the teacher to get right — that's not a bug, it's why you're reviewing them.

✓ Checkpoint: after a review pass, the Review queue should show no pending rows ("accepted: 120 · rejected: 40 · pending: 0" or similar). Pending and rejected rows never reach training — the next split silently skips them — so clear the queue before training.

Training configuration

Open Train. It is one form — Base model (+ Browse), Task, Seeds, Run name, a readiness line and the Train button (a confirm dialog shows the resolved config, then Launch); the knobs below live under More options → Model and hyperparameters (epochs, batch size, learning rate, Task Type) and LoRA and advanced training. The recipe defaults are sensible — for a first run, accept all of them:

Base model
HuggingFaceTB/SmolLM2-135M-Instruct. Small, instruction-tuned, runs on consumer hardware. The recipe trains the model to emit a JSON entity array as its output — the same LM head produces the structured output token-by-token, no separate extractive head is wired in. Alternative: Qwen/Qwen2.5-0.5B-Instruct for slightly better quality at 4x the size.
Adapter
LoRA, rank 16, alpha 32; leave target modules on auto (all linear layers for a model this size). Standard for small models. The structured-extraction adapter wraps each row's text + entities into a prompt + reference-completion pair so the model learns to generate well-formed JSON as its answer.
Learning rate
2e-4 for the LoRA. Standard for SmolLM2 on structured-output tasks; the model's small parameter count tolerates the higher rate.
Epochs
4. Span-extraction needs slightly more epochs than QA SFT because there are more output decisions per row (one per entity) and the head is fresh. Five epochs starts to overfit on small sets; three usually under-fits. (An empty field lets the app size epochs to your row count.) If you'd rather train an encoder tagger with exact offsets, set Task Type to Token classification (spans with exact offsets) and pick a DeBERTa / BERT-class base — see Tutorial 7.
Batch size + gradient accumulation
Batch 4, accumulate 4 → effective batch 16. Invoices can be 800-2,000 tokens, longer than classification inputs, so memory is more of a constraint. Adjust down on GPUs under 8 GB.

Expected runtime: 10-25 minutes on a single GPU (RTX 3060 or better), 30-60 minutes on CPU. The runs list under the form shows live loss and the validation metrics as sparklines, with a Cancel button on the running run; if F1 plateaus below 0.50 by epoch 2, cancel and check the answer key — it's almost always offset-drift in the key, not a model problem.

✓ Checkpoint: on Train, your run's row shows a live loss sparkline that drops from ~3-4 in the first few steps down to ~0.4-0.7 by the end. The validation span-set F1 climbs to ~0.70-0.85 by epoch 3. The bell shows the training Job with a percentage. When it completes, the automatic lift check scores the run and Results → Verdict shows the summary card (the lift against the base model; a token-classification run gets the span card with per-type precision / recall / F1 and the weakest types).

Read the trainability forecast

Before kicking off training, the trainability forecast (Train → More options → Plan and gaps, "Predicted gate-pass: ~N%"; the same number is the predicted_pass row of the goal ledger on Data → Sources → Details → Readiness overview) gives you a forecast based on row count, per-entity coverage, and base model size. For span-extraction specifically:

If the forecast is below 50%, training will pass the basic gates on the random-split test set but fail on the new-template held-out set. Spend the extra hour curating template-diverse gold instead of training. The single biggest predictor of a successful span extractor is template coverage, not model size.

Evaluation: span-set F1 + per-entity-type precision floors

After training, the automatic lift check scores the run and Results → Verdict shows the Pass/fail rules card (rule set, PASS/FAIL, the required rules that failed). The span-extraction scaffold ships four rules:

Span-set F1 ≥ 0.65
The headline. Precision = correctly-predicted entities / total predicted; recall = correctly-predicted entities / total in gold; F1 is the harmonic mean. An entity counts as "correct" only when type AND offsets both match the gold.
Span-set precision ≥ 0.70
Bias the model away from over-eager extraction. A model that predicts every numeric token as a candidate "total" scores high recall but low precision — and downstream that means human reviewers triaging false positives forever.
Span-set recall ≥ 0.60
Bias the model away from under-extraction. A model that only emits high-confidence vendor and total but skips line items scores high precision but low recall — and downstream that means line-item reconciliation breaks.
Safety pass rate (optional)
Catches refusal / off-topic / adversarial inputs that should be flagged through a different path. Useful when you wire the extractor behind a UI that surfaces uncertain extractions to a human.

The scorecard under the Pass/fail rules card lists every rule with its value and threshold so you see exactly which dimension is failing — "span-set F1 0.71 ≥ 0.65 passed, span-set precision 0.65 < 0.70 failed."

Why per-entity-type precision matters

The default span-set gates score the whole entity set in aggregate. For invoice extraction that's the wrong abstraction. Consider:

The platform's default scaffold ships a single uniform min_span_set_precision across all entity types. For invoice extraction you want tighter floors on specific entities — that means extra rules. The straightforward path: on Results → Failures, the Scaffolded pass/fail rule set card has + Add rule (the Metric dropdown lists the standard metrics and per-class groups once an eval has reported them); or copy the pack JSON, add per-entity rules that reference the per-entity precision values the span-extraction eval handler emits (precision.total, precision.invoice_date, etc.), and pick it in the eval-pack picker on the same tab in place of the default scaffold. Use the platform's /api/projects/<id>/evaluation/gates/<experiment_id> endpoint (visible in the API docs at http://localhost:8000/docs) to verify the new gates resolve against the eval result's metrics dict before you commit to them.

Don't gate every entity type at 0.95

It's tempting to set every per-entity-type precision floor to the strictest value. Don't. Tight floors on line items mean the rule set will fail a model that's perfectly good for the entities that matter, just because line items are inherently noisier. Set the floor where the downstream cost of a false-positive lives, not at the global maximum.

When the eval fails (common failure modes)

Span-extraction failure modes are different from classification or RAG failures. Common patterns and the fix for each:

SymptomRoot causeFix
Low recall on a rare entity type (e.g. due_date recall 0.40)Not enough gold examples for that type — model never learned the patternAdd 20+ answer-key rows that exercise the rare entity. Run Paraphrase positives seeded only from those rows.
Random-split test F1 strong (0.85) but new-template held-out F1 low (0.55)Label drift across templates — model memorised your top vendors' layoutsAdd gold from 3-5 new vendor templates. Re-stratify the split by template. Consider switching to a larger base model.
Overlapping spans in the prediction (one token tagged as both vendor and line_item)Ambiguous gold — same token tagged differently in different rowsSample 30 random gold rows; have a second reviewer re-tag without seeing the original. Disagreement > 15% on a single entity type = labeling problem. Reconcile and re-train.
Model emits a total for every invoice even when there isn't oneNo "no-total" examples in gold; model thinks every invoice must have a totalAdd 10-20 rows where the document is a statement / quote / partial invoice with no single total line. Use an empty total entity list (NOT a missing entry — explicit absence).
Span-set precision passes but the reroute recommendation on Results → Failures suggests a different recipeThe post-eval decision engine sees signals that suggest your task is shape-mismatchedRead the reroute trace under "Why this fired?". If it points at "qa-sft", you may be using span-extraction where a simpler answer-extraction would do.
Per-entity precision on total below 0.95 even after retrainingHard negatives in training: "subtotal", "tax", "shipping" not distinguished from "total"Run the Hard negatives playbook with a prompt targeting total-vs-subtotal look-alikes specifically. Review carefully — every accepted row teaches the model what NOT to grab as the total.

When the reroute recommendation on Results → Failures recommends an action, expand the "Why this fired?" disclosure on each signal. You'll see the actual numbers (matched_keywords, threshold deltas, per-entity-type scores) rather than just the recommendation verb. If the trace looks right, take the recommended action. If it looks wrong, the disclosure tells you exactly what the engine was matching on — file a bug and move on.

Ship as an inline microservice

Once the pass/fail rules pass (and you're happy with the per-entity-type precision on the entities that matter), ship in three steps:

  1. Export the model. Open Ship → Package → New export: pick the run, a Format (HuggingFace — a LoRA run is merged into its base at export, so the folder loads with plain transformers; GGUF (CPU) for llama.cpp / Ollama; ONNX, TensorRT-LLM, Docker), quantization and deployment targets, then Run Export. The export appears in the Releases table and under data/projects/<id>/exports/ with its manifest and release notes; Options → Compress merges and quantizes first if you want a smaller artifact.
  2. Deploy. Ship → Deploy holds the serve / deploy plans for the targets you ticked — a Docker export bundles a serve.py and Dockerfile, a GGUF export runs under the Ollama Runner target, Managed vLLM API is a plan you provision yourself — and Ship → Publish pushes a completed HuggingFace export to the Hub. Send the invoice text through whatever serves it (chat-completions for a vLLM / Ollama server); the trained model emits the entity JSON as its response. Wrap it in a thin /extract microservice that parses the JSON and enforces a schema before returning to the caller. Latency: 20-80 ms per invoice on a single GPU, 80-200 ms on CPU.
  3. Wire it into AP automation. Your AP ingest pipeline POSTs each new invoice text to the extractor microservice, gets back the entity JSON, and routes:
    • High-confidence extractions (all entities present, no entity-level confidence below the threshold you set) post straight to the ERP.
    • Mid-confidence extractions (a missing or low-confidence field) go to a human review queue with the highlighted spans pre-filled — your reviewer confirms or fixes, then posts.
    • Low-confidence or empty extractions (likely not an invoice, or an OCR failure) route to manual triage.

Smoke-test on Results → Try it first (/project/<id>/results?tab=try): Provider = Trained run (this project), pick the run — it loads in-process, no export needed. Paste 10 real invoices from production; check that each entity comes back with the right offsets and that the JSON output is well-formed. The per-turn footer (which run served the reply, the latency) is your sanity check before the model touches production traffic.

Inline doesn't mean autonomous

Even at 0.95 precision on total, a model will get one in twenty totals wrong. For a B2B finance team processing 10,000 invoices a month that's 500 wrong totals a month if every extraction posts straight through. Always wire a confidence threshold and a human review queue. Auto-post the high-confidence ones; review the rest. The extractor's job is to make AP staff faster, not to replace them.

What's next

You have a deployed invoice field extractor with per-entity precision floors and a sane human-review fallback. Three obvious next moves:

Extend to adjacent document types
The same recipe + workflow works for: purchase orders (extract vendor, PO number, line items, delivery date), contracts (parties, effective date, term length, governing law), bank statements (account, period, opening / closing balance, transaction rows), expense reports (employee, category, amount, date). New gold set; same training pipeline. The span-extraction recipe is more general-purpose than its name suggests — invoices are just the most common entry point.
Active-learning loop from the review queue
Every time a human reviewer corrects an extraction, that's a high-signal training example. Capture the corrections; add the confident ones to the training rows and the answer key; retrain when the key grows by ~50 rows. Over a quarter the model gets noticeably sharper on the templates your team actually receives, not the templates the public datasets happened to have.
Build a finance domain pack
Package the conventions you've established here — per-entity-type precision priorities, currency normalisation, the "explicit absence" gold convention — into a custom finance domain pack so your next AP extractor (POs, statements, expense reports) inherits the conventions without re-establishing them. Tutorial 9 — Building domain packs (finance / AP example) walks the canonical pack-construction process end-to-end with this exact use case as the worked example, including which pieces honestly belong in the pack vs the project's rule set.

Same recipe, different shape: Tutorial 7 — PII span tagging uses the same span-extraction machinery for open-set tagging instead of closed-set extraction (many PII spans per text, not exactly one of each entity type). The decision-tree differences are spelled out in T7's introduction; if you've made it here, T7 is the most natural follow-on.

From there, the tutorials hub links to other recipe shapes (classification, code review, RAG-protocol) and to the deeper Academy tracks — eval pack internals, synthetic-data tuning, the distillation workflow for shrinking your model further once the extraction quality is locked in.

Key terms

span-extraction recipe
BrewSLM recipe (task type Structured Span Extractor) that trains a small model to emit a JSON list of typed, offset-tagged spans from free-form text. Adapter default-canonical, task profile structured_extraction, scoring mode span_set.
Entity
One tagged span: a tuple of (type, character-start-offset, character-end-offset, raw-text). For invoices: vendor / total / line_item / invoice_date / due_date.
Span-set scoring
The eval handler compares prediction and gold as SETS of entities — F1 over (type, start, end) tuples. NOT field-match (which would require a single canonical field value per row). Per-entity-type precision and recall reported alongside the headline numbers.
Per-entity-type precision floor
An eval gate that requires precision on a specific entity type to meet a minimum threshold. For invoice extraction, set tighter floors on entities whose errors cost money downstream (total, dates) and looser floors on noisy entities (line_item).
Stratified-by-template split
Train/val/test split that puts different invoice templates (vendors, layouts) in each split. Random splitting overstates F1 because the model memorises layouts; stratified splitting forces the model to generalise.
Label drift
The same string tagged differently in different gold rows, usually because two reviewers disagreed or a single reviewer's standards drifted over the annotation session. The single biggest source of training-data noise in span-extraction projects.

Check yourself

Answers are saved to this browser.

← All tutorials