Tool
Is the lift real?
You scored the base model and the fine-tuned model on the same test rows, and the fine-tune came out ahead. Is that a real gain or the luck of which rows you tested on? Enter the counts and get the intervals, an exact test and a plain verdict, plus how many rows a smaller gain would need. Everything runs in your browser.
Calculator
Verdict
Fill in the form
What each number means
Accuracy and its interval
A score on n rows is a sample: test on a different n rows and you would get a slightly different number. The 95% interval is the range of true accuracies that fit what you saw. It uses the Wilson score interval, not the textbook p ± 1.96·√(p(1−p)/n), which is too narrow near 0% and 100% and has zero width when every row is right or every row is wrong (Brown, Cai and DasGupta 2001).
The change and its interval
The change is the fine-tuned accuracy minus the base accuracy, in points. Its interval is what decides the verdict. With the row-by-row counts it is a paired interval (Newcombe 1998, method 10): it knows that the two models were tested on the same rows, so rows both got right, or both got wrong, add no noise. Without them the tool can only treat the two scores as independent samples (Newcombe's unpaired hybrid score interval), which is wider: two models scored on the same rows almost always agree on most of them.
The exact McNemar test
Only the rows where the models disagree carry evidence about which is better. If the two models were equally good, each disagreement would be a coin flip. The p-value is the chance of a split at least as lopsided as yours under that coin flip (an exact binomial test, as McNemar's test is computed for small counts). A small p-value, conventionally under 0.05, says the split is unlikely to be luck. The exact form is a little conservative (Fagerland, Lydersen and Laake 2013). With very few disagreeing rows it is also the honest one: 3 rows all one way give p = 0.25 and 4 give 0.125, however the other rows fall, and it takes at least 6 all one way (p = 0.031) to get under 0.05. The paired interval can already sit above zero on 3 or 4 such rows, so the tool asks both to agree before it calls a lift.
The verdict
- Real lift: the whole interval for the change is above zero and, when you give the row-by-row counts, the exact McNemar p is under 0.05.
- Real drop: the same, below zero.
- Borderline: add rows: with the row-by-row counts, the interval and the exact test disagree. Usually the interval clears zero on a handful of changed rows while p is 0.125 or more: too few rows changed to rule out luck. Without the counts, the same applies when the two scores differ by 5 rows or fewer: even all of those rows changing one way could not give p under 0.05.
- Within noise: the interval includes zero. That does not mean "no effect"; it means this many rows cannot tell a gain from a loss.
- Too few rows: under 5 rows, where BrewSLM also gives no verdict.
BrewSLM's lift check puts a paired t-interval on the mean per-row change (+1, −1 or 0) instead; the
tool shows that interval too, so you can set it next to a BrewSLM run (BrewSLM reads its t value from a
rounded table above 30 rows, so the ends can differ in the last digit). On a few dozen rows the two
intervals differ by a point or two. The 50-row example in the box below is the same case as the first
validation row: its "−3 to +19" is BrewSLM's t-interval, the tool's paired interval is −2.7 to +18.3.
On its own the t-interval, like the paired interval, can clear zero on 3 or 4 changed rows all one way;
this tool does not call that a lift without the exact test, and neither does BrewSLM: when every row is
scored right or wrong, its lift check, Compare and Auto-RAG verdicts also need the exact McNemar p under
0.05 and report it as mcnemar_p. Where this tool says borderline, BrewSLM says
within noise and notes that too few rows changed (4 better, 0 worse on 50 rows: p = 0.125). For
scores with partial credit (F1, a judge score) BrewSLM keeps the t-interval alone.
How many rows you need
The smallest change a test set can reliably show, at 95% confidence and 80% power, is 2.80 × s_d / √n, where s_d is the standard deviation of the per-row change and 2.80 = 1.96 + 0.84. Turned around, a change of Δ needs about rows ≈ (2.80 × s_d / Δ)² paired rows. With 0/1 scores s_d² is about the share of rows that changed minus the squared change, so two models that agree on most rows need far fewer. Without pairing the figure is rows ≈ 2.80² × (p_base(1 − p_base) + p_tuned(1 − p_tuned)) / Δ² per model, usually several times more.
The habits that go with it
The same advice the academy gives next to every evaluation lesson; the interval is only as honest as the comparison behind it.
Validation
The site's tests run this page's own code on the cases below and compare every number with scipy 1.18
and statsmodels 0.15 on a CPU: the Wilson intervals with proportion_confint(method="wilson"),
the unpaired interval with confint_proportions_2indep(method="newcomb"), the McNemar p-value
with mcnemar(exact=True) and binomtest, the t-interval with
ttest_1samp. No library ships Newcombe's paired interval, so the reference writes it out from
statsmodels' Wilson limits and checks that it equals the unpaired interval when the two models are
independent (φ = 0). All agree to 10−6.
| Counts | Change, 95% interval (points) | McNemar p | Verdict |
|---|---|---|---|
| 50 rows: base 22, tuned 26; 6 better, 2 worse | −2.7 to +18.3 | 0.289 | within noise |
| 50 rows: base 23, tuned 27; 4 better, 0 worse | +0.6 to +15.1 | 0.125 | borderline |
| 50 rows: base 0, tuned 4; 4 better, 0 worse | −0.6 to +18.8 | 0.125 | within noise |
| 10 rows: base 0, tuned 4; no row-by-row counts | +3.8 to +68.7 | ≥ 0.125 | borderline |
| 5 rows: base 0, tuned 3; no row-by-row counts | +3.0 to +88.2 | ≥ 0.250 | borderline |
| 50 rows: base 23, tuned 29; 6 better, 0 worse | +3.0 to +20.4 | 0.031 | real lift |
| 5 rows: base 1, tuned 4; 3 better, 0 worse | +8.0 to +80.1 | 0.250 | borderline |
| 40 rows: base 24, tuned 29; 7 better, 2 worse | −1.7 to +26.0 | 0.180 | within noise |
| 78 rows: base 28, tuned 23; 2 better, 7 worse | −13.8 to +1.0 | 0.180 | within noise |
| 20 rows: base 0, tuned 0 | −16.1 to +16.1 | 1.000 | within noise |
| 20 rows: base 20, tuned 20 | −16.1 to +16.1 | 1.000 | within noise |
| 10 rows: base 0, tuned 10; 10 better | +60.8 to +100.0 | 0.002 | real lift |
| 1,234 rows: base 610, tuned 1,183; 580 better, 7 worse | +43.6 to +49.2 | < 0.001 | real lift |
| 200 rows: base 150, tuned 120; 10 better, 40 worse | −21.5 to −8.3 | < 0.001 | real drop |
| 5 rows: base 2, tuned 4; 2 better | −3.4 to +66.2 | 0.500 | within noise |
| 100 rows: base 50, tuned 50; 25 better, 25 worse | −13.6 to +13.6 | 1.000 | within noise |
| 300 rows: base 210, tuned 231; no row-by-row counts | −0.1 to +14.0 | — | within noise |
| 1 row: base 0, tuned 1 | −12.2 to +100.0 | 1.000 | too few rows |
The 300-row case is worth a second look: unpaired, a 7-point gain is just inside the noise. With the row-by-row counts it is a real lift whenever the models agree on most rows: 25 better and 4 worse gives +3.6 to +10.5 points (McNemar p < 0.001), and even 60 better and 39 worse gives +0.5 to +13.4 (p = 0.044). Collect the paired counts. The first 4-better, 0-worse row shows the other side: the interval clears zero, but four changed rows are too few for the exact test, so the verdict is borderline. (In the second, the interval itself already includes zero.)
Sources
- E. B. Wilson (1927), "Probable inference, the law of succession, and statistical inference", JASA 22:209; L. D. Brown, T. T. Cai, A. DasGupta (2001), "Interval estimation for a binomial proportion", Statistical Science 16:101.
- R. G. Newcombe (1998), "Interval estimation for the difference between independent proportions", Statistics in Medicine 17:873 (method 10), and "Improved confidence intervals for the difference between binomial proportions based on paired data", Statistics in Medicine 17:2635 (method 10).
- Q. McNemar (1947), "Note on the sampling error of the difference between correlated proportions or percentages", Psychometrika 12:153; M. W. Fagerland, S. Lydersen, P. Laake (2013), "The McNemar test for binary matched-pairs data", BMC Medical Research Methodology 13:91.
- E. Miller (2024), "Adding Error Bars to Evals", arXiv:2411.00640: paired differences and sizing for language-model evaluations.
Questions
Why should I pair the rows?
Because the two models answer most rows the same way. Rows both get right, or both get wrong, say nothing about which model is better, and the paired interval leaves them out of the noise. On the same counts it is usually much narrower than the unpaired one, so the same test set can show a smaller real gain.
Where do I get the row-by-row counts?
From the per-row scores of both runs: count the rows only the fine-tuned model got right and the rows only the base model got right. BrewSLM's Compare tab and lift check report them as better and worse rows; in your own script, zip the two lists of 0/1 scores and count.
Does "within noise" mean the fine-tune did nothing?
No. It means this many rows cannot tell a gain from a loss. The gain may be real and too small to see at this size. The sizing line says how many rows it would take; adding rows, or checking across training seeds, is the way to a clearer answer.
The interval and the McNemar p-value disagree. Which do I trust?
Neither on its own: the tool calls a lift or a drop only when both agree. They answer the same question in two slightly different ways, and the exact test is a little conservative. They disagree in two places: near p = 0.05, and when very few rows changed, where the interval can clear zero on 3 or 4 rows all one way while the exact p is 0.125 or more. The tool calls both "borderline" and asks for more rows; don't pick the answer you prefer.
Does a real lift mean the model is ready to ship?
No. A real lift says the fine-tune is better than the base model on this test set. Whether it is good enough is a separate rule you set before looking: a pass mark on the metric, per-class floors, no regressions on the cases that matter. Check that the test rows were never trained on, too.
Can I use this for F1, judge scores or other metrics?
Not directly: it assumes each row is right or wrong. For a score with partial credit, pair the per-row scores and put a paired t-interval (what BrewSLM's lift check uses) or a paired bootstrap interval on the mean per-row difference.
Related
- Lesson 1.18 (Evaluation that matches the task): baselines, gates and whether a difference is real.
- Lesson 2.7 (Evaluate by hand): score the base and the fine-tuned model on the gold set in code.
- FAQs: what "within noise" means in BrewSLM's verdict.
- Fine-tuning memory calculator: will the run fit before you start it?