Tool

Is the lift real?

You scored the base model and the fine-tuned model on the same test rows, and the fine-tune came out ahead. Is that a real gain or the luck of which rows you tested on? Enter the counts and get the intervals, an exact test and a plain verdict, plus how many rows a smaller gain would need. Everything runs in your browser.

Calculator

Scores on the same rows

Both models must be scored on these same rows, with the same answer key and scoring.

Row by row (optional, sharper)

Count the rows where only one model was right. With these the tool can pair the rows; without them it falls back to a wider, unpaired interval.

Sizing

At 95% confidence and 80% power, the convention BrewSLM's sizing uses.

Verdict

Fill in the form

Results, 95% intervals

    What each number means

    Accuracy and its interval

    A score on n rows is a sample: test on a different n rows and you would get a slightly different number. The 95% interval is the range of true accuracies that fit what you saw. It uses the Wilson score interval, not the textbook p ± 1.96·√(p(1−p)/n), which is too narrow near 0% and 100% and has zero width when every row is right or every row is wrong (Brown, Cai and DasGupta 2001).

    The change and its interval

    The change is the fine-tuned accuracy minus the base accuracy, in points. Its interval is what decides the verdict. With the row-by-row counts it is a paired interval (Newcombe 1998, method 10): it knows that the two models were tested on the same rows, so rows both got right, or both got wrong, add no noise. Without them the tool can only treat the two scores as independent samples (Newcombe's unpaired hybrid score interval), which is wider: two models scored on the same rows almost always agree on most of them.

    The exact McNemar test

    Only the rows where the models disagree carry evidence about which is better. If the two models were equally good, each disagreement would be a coin flip. The p-value is the chance of a split at least as lopsided as yours under that coin flip (an exact binomial test, as McNemar's test is computed for small counts). A small p-value, conventionally under 0.05, says the split is unlikely to be luck. The exact form is a little conservative (Fagerland, Lydersen and Laake 2013). With very few disagreeing rows it is also the honest one: 3 rows all one way give p = 0.25 and 4 give 0.125, however the other rows fall, and it takes at least 6 all one way (p = 0.031) to get under 0.05. The paired interval can already sit above zero on 3 or 4 such rows, so the tool asks both to agree before it calls a lift.

    The verdict

    BrewSLM's lift check puts a paired t-interval on the mean per-row change (+1, −1 or 0) instead; the tool shows that interval too, so you can set it next to a BrewSLM run (BrewSLM reads its t value from a rounded table above 30 rows, so the ends can differ in the last digit). On a few dozen rows the two intervals differ by a point or two. The 50-row example in the box below is the same case as the first validation row: its "−3 to +19" is BrewSLM's t-interval, the tool's paired interval is −2.7 to +18.3. On its own the t-interval, like the paired interval, can clear zero on 3 or 4 changed rows all one way; this tool does not call that a lift without the exact test, and neither does BrewSLM: when every row is scored right or wrong, its lift check, Compare and Auto-RAG verdicts also need the exact McNemar p under 0.05 and report it as mcnemar_p. Where this tool says borderline, BrewSLM says within noise and notes that too few rows changed (4 better, 0 worse on 50 rows: p = 0.125). For scores with partial credit (F1, a judge score) BrewSLM keeps the t-interval alone.

    How many rows you need

    The smallest change a test set can reliably show, at 95% confidence and 80% power, is 2.80 × s_d / √n, where s_d is the standard deviation of the per-row change and 2.80 = 1.96 + 0.84. Turned around, a change of Δ needs about rows ≈ (2.80 × s_d / Δ)² paired rows. With 0/1 scores s_d² is about the share of rows that changed minus the squared change, so two models that agree on most rows need far fewer. Without pairing the figure is rows ≈ 2.80² × (p_base(1 − p_base) + p_tuned(1 − p_tuned)) / Δ² per model, usually several times more.

    The habits that go with it

    The same advice the academy gives next to every evaluation lesson; the interval is only as honest as the comparison behind it.

    Validation

    The site's tests run this page's own code on the cases below and compare every number with scipy 1.18 and statsmodels 0.15 on a CPU: the Wilson intervals with proportion_confint(method="wilson"), the unpaired interval with confint_proportions_2indep(method="newcomb"), the McNemar p-value with mcnemar(exact=True) and binomtest, the t-interval with ttest_1samp. No library ships Newcombe's paired interval, so the reference writes it out from statsmodels' Wilson limits and checks that it equals the unpaired interval when the two models are independent (φ = 0). All agree to 10−6.

    Cases checked, including 0 and n
    CountsChange, 95% interval (points)McNemar pVerdict
    50 rows: base 22, tuned 26; 6 better, 2 worse−2.7 to +18.30.289within noise
    50 rows: base 23, tuned 27; 4 better, 0 worse+0.6 to +15.10.125borderline
    50 rows: base 0, tuned 4; 4 better, 0 worse−0.6 to +18.80.125within noise
    10 rows: base 0, tuned 4; no row-by-row counts+3.8 to +68.7≥ 0.125borderline
    5 rows: base 0, tuned 3; no row-by-row counts+3.0 to +88.2≥ 0.250borderline
    50 rows: base 23, tuned 29; 6 better, 0 worse+3.0 to +20.40.031real lift
    5 rows: base 1, tuned 4; 3 better, 0 worse+8.0 to +80.10.250borderline
    40 rows: base 24, tuned 29; 7 better, 2 worse−1.7 to +26.00.180within noise
    78 rows: base 28, tuned 23; 2 better, 7 worse−13.8 to +1.00.180within noise
    20 rows: base 0, tuned 0−16.1 to +16.11.000within noise
    20 rows: base 20, tuned 20−16.1 to +16.11.000within noise
    10 rows: base 0, tuned 10; 10 better+60.8 to +100.00.002real lift
    1,234 rows: base 610, tuned 1,183; 580 better, 7 worse+43.6 to +49.2< 0.001real lift
    200 rows: base 150, tuned 120; 10 better, 40 worse−21.5 to −8.3< 0.001real drop
    5 rows: base 2, tuned 4; 2 better−3.4 to +66.20.500within noise
    100 rows: base 50, tuned 50; 25 better, 25 worse−13.6 to +13.61.000within noise
    300 rows: base 210, tuned 231; no row-by-row counts−0.1 to +14.0—within noise
    1 row: base 0, tuned 1−12.2 to +100.01.000too few rows

    The 300-row case is worth a second look: unpaired, a 7-point gain is just inside the noise. With the row-by-row counts it is a real lift whenever the models agree on most rows: 25 better and 4 worse gives +3.6 to +10.5 points (McNemar p < 0.001), and even 60 better and 39 worse gives +0.5 to +13.4 (p = 0.044). Collect the paired counts. The first 4-better, 0-worse row shows the other side: the interval clears zero, but four changed rows are too few for the exact test, so the verdict is borderline. (In the second, the interval itself already includes zero.)

    Sources

    Questions

    Why should I pair the rows?

    Because the two models answer most rows the same way. Rows both get right, or both get wrong, say nothing about which model is better, and the paired interval leaves them out of the noise. On the same counts it is usually much narrower than the unpaired one, so the same test set can show a smaller real gain.

    Where do I get the row-by-row counts?

    From the per-row scores of both runs: count the rows only the fine-tuned model got right and the rows only the base model got right. BrewSLM's Compare tab and lift check report them as better and worse rows; in your own script, zip the two lists of 0/1 scores and count.

    Does "within noise" mean the fine-tune did nothing?

    No. It means this many rows cannot tell a gain from a loss. The gain may be real and too small to see at this size. The sizing line says how many rows it would take; adding rows, or checking across training seeds, is the way to a clearer answer.

    The interval and the McNemar p-value disagree. Which do I trust?

    Neither on its own: the tool calls a lift or a drop only when both agree. They answer the same question in two slightly different ways, and the exact test is a little conservative. They disagree in two places: near p = 0.05, and when very few rows changed, where the interval can clear zero on 3 or 4 rows all one way while the exact p is 0.125 or more. The tool calls both "borderline" and asks for more rows; don't pick the answer you prefer.

    Does a real lift mean the model is ready to ship?

    No. A real lift says the fine-tune is better than the base model on this test set. Whether it is good enough is a separate rule you set before looking: a pass mark on the metric, per-class floors, no regressions on the cases that matter. Check that the test rows were never trained on, too.

    Can I use this for F1, judge scores or other metrics?

    Not directly: it assumes each row is right or wrong. For a score with partial credit, pair the per-row scores and put a paired t-interval (what BrewSLM's lift check uses) or a paired bootstrap interval on the mean per-row difference.