Teaching a 1.5B model to read Pub 17: two misses, one pass

The result, in one table

157 held-out questions, the same retrieval for every arm (top-3 passages of 600 characters, no reranker), the same judge (gemma4:12b running locally, grading each answer against the answer key as correct / partial / wrong = 1 / 0.5 / 0):

Paired row by row against the bar, run #162 is better on 32 questions, worse on 12 and the same on 113: +0.118, 95% interval +0.054 to +0.182. The weakest seed cleared it by much less (+0.067, interval +0.003 to +0.131), and averaged over the three seeds the gain is +0.091 (+0.028 to +0.155). So read the headline as "about +0.07 to +0.12, depending on the seed", not as exactly +0.118. The spread between seeds is about as large as the weakest seed's gain.

The no-passages line matters as much as the headline. On its own the reader scores 0.185: it did not memorise the publication, and it isn't supposed to. Retrieval still does the remembering, which is part 1's result. The fine-tune taught the model to read.

Why the bar is the base model with passages

A fine-tune that answers from passages is easy to make look good: compare it with the base model without passages and you get 0.07 → 0.69, a tenfold "improvement" that retrieval alone produces. The honest comparison is the base model reading the same passages for the same questions, scored by the same judge. That's the bar. The fine-tune has to beat it beyond row-to-row noise, or it hasn't earned its training run.

We also wrote the rules down before anything was scored:

• The test and validation sets were frozen and checksummed first. 157 questions each, never edited afterwards, with no question shared between train, validation and test, and no source passage shared between test and validation.
• Three seeds on identical data. One training run is one draw. The same comparison across three seeds is how you find out whether a gain is the method or the luck of the draw.
• The seed to publish is picked on validation, never on test. The pass rule: the picked seed must be better than the bar on test, with the 95% interval above zero, and at least two of the three seeds must be better too.

What changed from part 1's plan, and why

Part 1 said the reader would be judged on the same 41 questions, against base + passages at 0.73. Two things changed that plan.

• The model. Qwen2.5-3B-Instruct, part 1's base, has a research-only licence, so a fine-tune of it couldn't be published. Qwen2.5-1.5B-Instruct is Apache 2.0. The smaller model is also a weaker reader, at 0.585 with passages on the same 41 rows against the 3B's 0.73, which leaves more room for training to help.
• The test set. The first 1.5B reader trained without the abstention rows described below (run #153) scored 0.732 against the 1.5B bar of 0.585 on those 41 rows. That was better, but the 95% interval was +0.004 to +0.289. 41 rows can't tell +0.05 from +0.25. Part 1's retrieval had also been chosen on those same 41 rows. So the 41 rows were retired as the judge of anything, the retrieval was re-chosen on a separate validation set, and every later number in this post is on 157 fresh test questions spread over the whole publication.

The very first reader is worth a line too. Run #152 trained on the pairs' own answers, with the right passage held out of 20% of prompts so the model would learn to say "I don't know". It learned that too well: on validation it said "the passages don't say" on 35% of questions whose answer passage was right there in the prompt. The cause was in the data. A pair's answer was written from one passage, but retrieval doesn't always return that passage, so the targets disagreed with the prompts. That led to the fix every later attempt uses.

The recipe: training equals serving

Every training question is retrieved with the project's own retrieval and rendered exactly as serving will send it: the numbered passages in the system message, then the question. A teacher model answers from those same passages, citing [1], [2], …, and its answer becomes the training target. When the passages don't hold the answer, the teacher says so, and that answer is kept as a natural "I don't know". No rows are withheld artificially. A local judge then grades every teacher answer against the pair's reference answer and drops the ones it rejects, before anything trains.

In BrewSLM that's the Documents → Q&A card on Train, with Train the model to answer from your passages and Use a teacher model's answers to the same passages as the targets ticked, and the teacher picked in the Generation model picker under it.

Attempt 1: one writer cleared it, the other didn't

The first full attempt ran the recipe twice on the same 157-question test set (bar 0.621, base + passages), changing only the model that wrote the training data, meaning the question → answer pairs and the teacher answers:

The stronger writer beat the bar on two of three seeds. Every seed from the open-weight writer scored above the bar, but none beyond noise. A flagship model can be used for higher quality, and here it measurably was. But we publish weights only when every row of their data, test set included, came from an open-weight model we can name on the model card. So the question became: what was the open-weight pipeline getting wrong?

The data said two things:

• Questions that point at nothing. 292 of the open-weight writer's answer-key questions were thrown out for asking about "this passage" instead of naming their subject. A question like that can't be answered by anyone who hasn't seen the passage, so retrieval can't find it and the model can't answer it. About 9% of its training questions had the same habit, and those weren't caught.
• Answers the judge rejected, and partial answers kept. The judge rejected 12.7% of the open-weight teacher's answers, against 3.9% for the flagship's. The filter dropped only answers graded wrong. Answers graded partial trained.

What we changed in BrewSLM

Each failure became a general option rather than a one-off script:

• Question repair. The flow now rewrites a generated question that points at "the passage" or "this section" so it names its subject, or drops it when it can't. On attempt 2 it rewrote 60 and stripped 9, and the answer-key questions thrown out for pointing at nothing fell from 292 to 4.
• A stricter keep rule for teacher answers. teacher_filter_keep: "correct" keeps only answers the judge grades fully correct. The default, "not_wrong", still keeps partial ones. It's an API option on the flow today, and the card doesn't show it yet.
• Critique and revise. Instead of throwing away every rejected answer, Revise answers the checker rejects (one round of feedback) sends each one to a critic. The Critic model picker chooses it, or the judge's own reason is used at no extra call. The critic reads the question, the reference, the answer and the passages, and says what's wrong. The teacher rewrites the answer from the same passages, and the same judge grades the rewrite and has the final say. We deliberately used a critic that isn't the evaluation judge, so the training answers aren't tuned to one model's taste.
• A run that dies resumes. Generation replies, teacher answers and judge verdicts are saved as they arrive. Attempt 2's teacher step was stopped at the filter for a machine restart, then re-posted with the same request, and it picked up at judged answer 1,650 of 2,890 without asking the teacher anything again.

Attempt 2: all open weights, the test set included

A new project, built from scratch from the same PDF: same cleaning, same base model, same retrieval. gpt-oss:120b wrote everything: questions, reference answers, the 157-question test set, the 157-question validation set, and the teacher answers. gemma4:12b judged. This test set is not attempt 1's, which is why the bar moves from 0.621 to 0.570, and why the two attempts' scores aren't directly comparable. Each is only meaningful against its own bar.

Then the rule, applied mechanically. Validation scores were 0.710, 0.704 and 0.723 for seeds 42, 43 and 44, so seed 44 is the pick. Its test result is better than the bar with the interval above zero (+0.054 to +0.182). All three seeds are better, and the rule needed two. Pass.

One thing we found while writing this up: the flow's own summary said the revise round "attempted 628" but the filter "dropped 317", which don't reconcile. The missing 69 were revisions that came back as "I don't know". They were correctly kept out of training, but they dropped out of every count. That's fixed: the counts now add up, and the 69 have a line of their own.

What the number does and doesn't say

• Retrieval is the ceiling. The retriever found the right passage for 123 of 157 test questions, the same for every arm. 28 of the published model's 41 wrong answers are questions where the right passage was never in the prompt. No reader can fix those. The other 13 are misreads: the right passage was there and the model got it wrong. The base model misread 29.
• Same family, in distribution. The test questions were written by the same model that wrote the training questions. The result says this reader handles that kind of question about this document better than its base. It says nothing about questions people would phrase differently, or about tax knowledge in general.
• One judge. gemma4:12b filtered the training answers and scored the test, so its preferences shape both. A different judge might rank the arms differently.
• Validation ran higher than test. The fine-tunes scored 0.70 to 0.72 on validation and 0.64 to 0.69 on test, while the base bar was lower on validation (0.551 vs 0.570). We used validation only to pick the seed, never to report the result.
• It cites in two formats. 505 of the 2,779 training answers kept the teacher model's own citation style, 【1†L1-L4】 (passage 1, lines 1-4), because our cleanup step converted 【1】 to [1] but missed that form. The trained model copied it: 73 of its 157 test answers cite that way, and 66 use [1]. The number is still the passage number. The judge grades facts, so the scores don't depend on it, but a strict parser would. The cleanup step now handles both forms, also for answers already cached, so the next training run won't learn it. This model keeps the habit, and its card says so.
• The project's default pass/fail rules say FAIL. BrewSLM's generic rule set for this task wants token-F1, an LLM-judge pass rate and a safety pass rate, none of which this run measured, so the scorecard reports them as failed or missing. We judged this model on the rule we wrote down before scoring, not on those defaults. We mention it because a screenshot of that panel would show a red FAIL, and it should.

Try it

The model is TensorGreed/brewslm-pub17-passage-reader (Apache 2.0, merged weights, about 3 GB). Its model card has the exact prompt layout it was trained on, a working transformers snippet, every number above, and how the data was made. Put it behind a retriever over Publication 17, keep the passages in the system message, and it cites the passage it used (as [1] or 【1†L1-L4】, see above) or says it doesn't know.

The same run from a script

Every option above is a field on one request. Against a running BrewSLM:

# pairs, answer key, split, teacher answers, filter, one revise round, then train
POST /api/projects/{id}/flows/documents-to-qa   {
  "backend": "ollama:gpt-oss:120b",
  "train": true,
  "answers_from_teacher": true,  "teacher_backend": "ollama:gpt-oss:120b",
  "teacher_filter": true,        "teacher_filter_keep": "correct",
  "teacher_revise_rounds": 1,    "teacher_critic_backend": "ollama:gpt-oss:120b",
  "keep_test": true,             "decision_gap": 0.08,  "concurrency": 4
}
# three seeds of the same run's exact config, as one seed group with its own lift check
POST /api/projects/{id}/evaluation/summary/check-seeds   {"experiment_id": {run}, "num_seeds": 3}
# score any finished run on validation, as a tracked Job
POST /api/projects/{id}/auto-rag/comparison/run?corpus=documents&split=val&experiment_id={run}

Publication 17 is a work of the United States government and in the public domain. The run ids in this post (#160, #161, #162) are the real ones.