Narrative in, JSON out: a 1.5B accident-report extractor that knows when to say null

The result, in one table

1,234 held-out test narratives. Every output is parsed as JSON and compared field by field with the answer key. The comparison is typed: numbers as numbers, strings ignoring case and spacing, and null, a missing key and "" all count as null.

* A 300-row subset of the same test set, run by a separate script and scored with BrewSLM's own scorer. It gives the base model its chat template, every field's type and allowed values, and "use null when the narrative doesn't state it". We ran it so the fine-tune couldn't be credited with what a better prompt would have done anyway. Prompting fixed the format (0.042 → 0.770 schema-valid), but accuracy didn't move, and the base model started inventing: guessed dates, "Not specified in the narrative" as an aircraft make, values for fields the text never mentions. Null recall fell, from 0.545 to 0.371. Telling a small model "don't guess" isn't the same as teaching it not to.

Why this task, and why null is the point

Extraction is the most common thing people want a small model for that isn't chat: read a document, fill in a form. The NTSB's accident database is a good test bed. It's public domain, it's large, and every event has both a narrative written by an investigator and a set of coded fields filled in by hand. That looks like a ready-made answer key. It isn't, and that's what this project was about.

The coded fields come from the whole investigation: forms, records, interviews. The narrative is a summary, and it often doesn't repeat them. In the 12,421 events we used, the narrative names the aircraft make in 40% of cases and states the pilot's total hours in 7%. Train a model to output the coded value from the narrative, and for those rows you are training it to produce a fact that isn't in its input. That is invention, taught on purpose. A model trained that way will fill in the pilot's hours for any narrative you give it.

So the answer key is masked. For each field we start from the investigators' coded value, and keep it only if a fixed, deterministic text check finds it stated in the narrative. Otherwise it's null. The make or model has to appear as written. The injury counts have to be stated. The date has to appear in the text: when the text's date is a day off the coded one, the text wins, because the model can only learn to read what's there. No language model touched the labels, training or test. The masking script went through three rounds of fixes on a 50-row hand audit. Its first version, for example, accepted the bare word "air" as a mention of the make Air Tractor.

That gives the score a specific meaning. It measures extracts what the text says, not knows what happened. It is also slightly conservative. A check can miss a value phrased in an unusual way, and the model is then marked wrong for extracting it.

The rule, written down first

Before training we froze the rule and a checksum of it. Three seeds (13, 42, 1234) train on identical data. Each seed is evaluated on test once, and every gate must pass on every seed:

• The 8 fact fields are the ones a narrative states directly: make, model, date, state, the three injury counts and pilot hours. The other five (aircraft category, damage, highest injury, operating rule, flight purpose) map free wording onto fixed codes. We report them but didn't gate them per field. We couldn't justify a floor for them before seeing a trained model.
• Null recall matters most for safety. For an extractor, the harmful mistake is inventing a value the text never states. Under a masked answer key, that's exactly what null recall measures, so it has two floors.
• The rule was amended twice, before any evaluation ran. The per-row gain uses per-row field F1, because BrewSLM's evals store that per row but not per-row exact match. And we hadn't said which seed to publish, so we fixed it as the best on validation. Both changes are in the frozen file with their dates.
• One test look means one. BrewSLM runs an automatic 100-row test check after every training run. We turned it off for this project, so each seed's single full test eval is the only time test was scored.

Validation scores were 0.9595, 0.9614 and 0.9610 for seeds 13, 42 and 1234, so seed 42 is the published one. All three seeds passed every gate. Pass. The seeds agree to within 0.002 on field accuracy, so this isn't a lucky draw.

Where it still fails

• Aircraft model is the big miss. 454 test narratives state the model designator. The model returned null for 232 of them and a different designator for 46. "Beech 58", "Bell 47-G", "Cessna 152": it gets the make and drops the model. Most of the gap between 0.959 field accuracy and 0.625 whole-record accuracy is this one field. If you need the model designator, check its nulls against the text.
• Its mistakes lean toward null. Across all fields it wrote null for a stated value 382 times, and invented a value for an unstated field 192 times. For an extractor that's the safer direction, but it's why null precision (0.954) only just clears its floor of 0.95. About 1 in 22 nulls hides a value the text does state.
• The answer key's own blind spots. Some of those "missed" models may be the masking check's fault, not the model's: if training rows nulled designators the check failed to recognise, the model learned some of that caution. We didn't measure how much. It's the first thing to look at for a v2.

What BrewSLM needed for this

Three things BrewSLM couldn't do at the start, each built as a general feature rather than for NTSB:

• A JSON-extraction recipe with typed scoring. Extraction used to be scored by comparing each field as a string, with no idea of types, nulls or allowed values. The new json-extraction recipe takes an output schema (types, enums, nullable fields) and scores against it. Numbers compare as numbers. Null precision and recall are computed per field. Out-of-vocabulary enum values and type violations get their own rates. "Valid JSON" and "valid against the schema" are separate numbers, as the first table shows they should be. The Data step has a schema editor, and Results shows the per-field table.
• Bring your own train / validation / test. BrewSLM always split a dataset itself. Here the split had to be by NTSB event id, decided outside, with 1,234 test rows fixed in advance. Use my own train / val / test files on the Data split form imports three files with exact membership: no shuffle, no ratio, no dedup. It refuses (or, if you ask, drops from train) any training row whose input also appears in validation or test. It reports rejected rows by file and line, and no later step (cleaning, synthetic data, re-splitting) can move a test row into training. That last guarantee took several rounds of independent code review to get right. Round after round found another way a test row could leak back in: a re-split, a malformed line, a column that looked like the input but was really the label.
• Rows whose input is their own answer. One of those reviews found that a badly mapped dataset could train a model on input = label and look perfect. BrewSLM now rejects such rows by name (input_is_answer, classification_input_is_label) instead of training on them, in the import and in the normal split.

Training

One seed didn't start cleanly. Seed 1234 failed two minutes in, before its first training step, on a DNS error reaching the model hub. No eval had run, so we restarted it with the same config. We mention it because the run log shows a FAILED row, and it should.

Try it

The model is TensorGreed/brewslm-ntsb-accident-extractor (Apache 2.0, merged weights, about 3 GB). Its card has the exact prompt, a working transformers snippet, the schema with every allowed value, and every number above. Before publishing, we re-ran that snippet against the merged weights on 100 test narratives. It parsed all 100 outputs and scored 0.976 field accuracy on that sample.

The same run from a script

Against a running BrewSLM, the whole project is a handful of requests:

# recipe + output schema (types, enums, nullable fields)
PUT  /api/projects/{id}/recipe                 {"recipe_id": "json-extraction"}
PUT  /api/projects/{id}/recipe/output-schema   {"output_schema": { ...13 properties... }}
# your own splits, exact membership; refuse any train row that leaks into val/test
POST /api/projects/{id}/dataset/predefined-split   train=@train.jsonl val=@val.jsonl test=@test.jsonl on_leak=fail
# three seeds of one config, as a seed group
POST /api/projects/{id}/training/experiments   {"name": "...", "config": {
  "base_model": "Qwen/Qwen2.5-1.5B-Instruct", "use_lora": true,
  "max_seq_length": 4096, "seeds": [13, 42, 1234]}}
# one full held-out eval per seed, greedy
POST /api/projects/{id}/evaluation/run-heldout?async_job=true   {"experiment_id": {run},
  "dataset_name": "test", "max_samples": 5000, "max_new_tokens": 512, "temperature": 0}

The NTSB data is U.S. government work in the public domain. Courtesy: National Transportation Safety Board. The answer key is a masked derivative of the NTSB's coded fields, not NTSB findings, and this project isn't affiliated with or endorsed by the NTSB. NTSB accident reports can't be used as evidence in a civil action for damages (49 U.S.C. § 1154(b)), and nor should a model trained on them.