Build a code review nitpicker with the code-review recipe

This is the BrewSLM workflow for any project where the model needs to read code and write a comment about it in someone's voice — PR review, internal lint-explanation, style-guide enforcement, security-review note generation, deprecation notice writing. The recipe stays the same; only the gold set's voice changes. Train one model per team to inherit that team's review culture.

The end state is the model you'd actually deploy alongside your existing CI: small enough to run on a self-hosted runner, fast enough to comment within seconds of the PR opening, and tuned to your team's conventions in a way no off-the-shelf tool can match.

What you'll build

A code-review model that takes a unified-diff hunk as input and produces 1-3 review comments as output. Concretely:

Input:

@@ -42,6 +42,11 @@ def handle_request(req):
     if not req.user:
         return error_response(401)

+    if req.payload == None:
+        return error_response(400)
+
+    logger.info(f"Processing request for {req.user.email}")
+
     result = process(req.payload)
     return success_response(result)

Output:

1. Use `is None` instead of `== None` for the null check — `==` calls __eq__ which can be overridden and is slower.
2. Log line leaks user email into INFO-level logs. Either drop the email or move to DEBUG to stay GDPR-compliant.
3. Consider whether a 400 is the right code here — empty payload may indicate a client retry; 422 reads better.

The model is a fine-tuned LoRA adapter on top of Qwen/Qwen2.5-Coder-1.5B-Instruct (the recipe's suggested base). Latency: 1-3 seconds per diff on a single GPU; the bigger Qwen2.5-Coder-3B-Instruct alternative reads more context and writes more nuanced reviews at 3-6 seconds. Deployable as a GitHub Action that runs on PR open / PR update.

Why a small model (not GPT-4, not regex)

Three options exist for automated PR review. Use this comparison:

Lint catches what you can encode as rules; nobody has a regex that captures "this looks like a feature flag check but the flag is already removed from config" or "the test name implies behaviour the test doesn't verify". Frontier models are good at language but their latency makes them awkward in CI, their cost compounds at high PR volume, and shipping your proprietary code to a third-party API is a non-starter for many enterprise deployments. Small fine-tuned models occupy the gap: fast enough for CI, private enough for proprietary code, and — crucially — voice-tunable in a way no off-the-shelf tool is.

Choose your dataset

You need (diff, review) pairs. The diff is a unified-diff hunk (the bit between @@ -start,len +start,len @@ markers); the review is the comment text a human reviewer would write. Four sources, in order of value:

Your own internal PR review archive (by far the highest value)

Mine 6 months of merged PRs from your team's GitHub via gh api repos/<org>/<repo>/pulls?state=closed + /pulls/<n>/comments. Filter for inline comments on diffs (not top-level PR comments). This captures voice + conventions a public dataset never will. A 50-engineer team typically produces 2-5k inline review comments per month — 6 months gives you 12-30k candidate rows, easily enough.

The CodeReviewer dataset (warmup / pretraining)

Microsoft Research's CodeReviewer paper released a corpus of millions of GitHub PR reviews across languages. Useful as a warmup pass before fine-tuning on your own data — gets the model speaking "review language" generically before you teach it your voice.

CodeSearchNet (unlabeled diffs)

The CodeSearchNet code corpus is unlabeled function-level code in 6 languages. Useful if you need additional diff context for the synthetic playbooks but don't have enough archive of your own. Pair with a teacher model to generate review-like comments as a bootstrap step.

Open-source merged PRs (for solo devs / new teams)

If your team is too new to have a meaningful archive, mine merged PRs from large open-source projects (Linux kernel, Postgres, React, Django, Rust compiler). Public, plentiful, and the reviews are typically high-quality. Trade-off: you inherit the open-source project's voice, not yours.

Ingest and map

In BrewSLM, create a new project: Projects → + New Project, give it a name and describe the job in What problem do you want this model to solve? ("read a code diff and write 1-3 review comments in our team's voice"). The project opens on its Overview. Then pick the recipe: Train → More options → Task type → the Code Review Nitpicker card → Use this task type. It sets the adapter (qa-pair), task profile (instruction_sft — Data → Labels → What kind of task is this? will read Instruction following), scoring mode (field_match), and the code-review rule set (LLM-judge headline rule, F1 as a sanity check).

Open Data → Sources → Import dataset (auto-mapping) → Import dataset →. The wizard walks Source → Task type → Map → Preview & Confirm; point it at your CSV or JSONL. The expected shape is two required fields plus an optional rationale:

{
  "diff": "+ if (user.id == null) {\n+     return;\n+ }",
  "review": "Use `===` for null check, or `if (user.id != null)` to also catch undefined.",
  "rationale": "JS equality nit; the loose-equality form coerces both null and undefined."
}

The wizard's Map step scans the columns and matches them against the recipe's shape signature (input columns named like diff/patch/code/file/snippet, output columns like review/comment/feedback/suggestion). Preview & Confirm shows the accepted sample rows; click Import N row(s) to project when the preview looks right.

✓ Checkpoint: Data → Sources lists the import (Details → Sources summary shows the row count per source). The goal ledger on the Readiness overview (same Details drawer) turns its data_ready row green once the row count crosses 100; below that it shows amber with a "more data recommended" hint. If the mapping is wrong, the preview rows will be empty or mis-shaped — go Back to the Map step and pick the correct columns before importing.

Cleanup

Open Data → Prepare and its Details → Quality and safety section. For code-review data, the cleanup pass is mostly about noise removal:

• Strip auto-generated comments. Your archive will include CI-bot comments ("✅ Build passed", "Codecov: -0.2%"), Dependabot dependency-bump notices, Snyk security-scan summaries, copilot suggestions accepted-as-comments. None of these are reviews; all of them are noise. Filter by author handle (drop rows where commenter ends in [bot]) and by content heuristics (drop rows starting with markdown badge images).
• Drop LGTM-only rows. "LGTM", "ship it", "👍", "approve" — these are signal-zero for training a model to write reviews. They're useful as hard negatives (see the synth section below) but should not be in your positive gold set.
• Normalise diff whitespace. Mixed line endings (CRLF vs LF), trailing whitespace on diff lines, smart-quotes in review text. The near-duplicate signal in Data health and leakage (Data → Split → Details) will catch many of these, and Prepare's one-click autofixes apply the fix — review one cluster at a time.
• Deduplicate near-identical reviews. The same engineer often writes the same review on similar diffs ("add a docstring", "extract this magic number"). Keep one canonical instance per pattern — train/test leakage on duplicate reviews inflates F1 scores artificially.
• PII in commit history. Reviews sometimes quote internal Slack messages, customer names, or production credentials accidentally pasted in. Walk the Quality and safety section's PII flags and decide per-row (Prepare's Redact PII & Secrets checkbox redacts per field on Clean All Documents).

Skip nothing here. The code-review model inherits whatever signal you don't filter — if half your training rows are bot comments, the model will learn to write bot comments.

Pick the recipe: code-review or qa-sft or classification?

BrewSLM ships several recipes that could plausibly handle review-style tasks. Use this decision tree:

For the canonical "comment on diffs in your team's voice" use case: code-review wins because (a) the LLM-judge eval handles the "many valid reviews" problem that breaks F1-led recipes, (b) the scaffold expects a diff input + review output without forcing you to invent mapping, and (c) the three code-review playbooks target voice, anti-patterns, and failure clusters specifically. Sticking with code-review for the rest of this tutorial.

Domain packs (the language-specific gap)

BrewSLM doesn't ship a code-review domain pack today — the curated packs (legal, support, ecommerce, healthcare; Settings → Tools → Domain Packs) are content domains, not engineering domains. For code-review you're operating on the recipe defaults, which is fine for v1.

A worthwhile follow-up project is to build a custom pack that bundles your team's style guide as prompt context — the synth playbooks accept extra context, so a style_guide.md excerpt injected into the playbook prompt teaches the synth-generated reviews to align with your conventions. The pack would also bundle: cleaning rules tuned to your CI bot author handles, an LLM-judge rubric that calls out your specific conventions ("does the review match our naming convention for boolean flags?"), and language-specific severity ladders ("nit / suggestion / blocker" for JS vs "info / warn / error" for Go).

If you're shipping this to a security team or to multiple language verticals (one model per language), packaging the conventions as a domain pack pays back fast. The platform's pack framework is general-purpose; the recipe is one input among many.

Build the answer key (gold set) — exemplary reviews + LLM-assisted promotion

The answer key — BrewSLM's name for the gold set — is where the model's voice comes from. Two complementary paths, both worth running:

Path A — manual seeding from "exemplary" reviews

Pick your 3-5 senior engineers (the ones whose reviews other engineers learn from). Walk their last 50 PR reviews each. Grab the inline comments where:

• The reviewer raised a real concern with a concrete suggestion.
• The PR author acknowledged the comment (replied "good catch" / "fixed in next push" / made the change).
• The comment is 1-3 sentences — long enough to explain, short enough to read in a glance.

Open Data → Labels → the Answer key card (Practice answer key / Final answer key picker, Lock). For each row paste the diff context (4-6 lines around the comment line) into the input field, paste the comment verbatim into the expected-answer field, and add a one-line rationale tag (style-nit, bug-suggestion, test-gap, convention, security). Spend 60 minutes here. These ~50-100 rows are the most important data in the project — they imprint voice on the rest.

Path B — LLM-assisted promotion from your full archive

Your full PR archive is too large to hand-curate. For the bulk pass, an out-of-band scoring step works well — the platform doesn't bundle a "rank by quality" UI, so you'll script this against your teacher backend directly:

1. Bulk-import your filtered PR review archive (post-cleanup) into the project.
2. Write a short script that POSTs each (diff, review) pair to your teacher (Ollama / OpenAI / Anthropic — whichever you've set up as a synth backend). Prompt the teacher to score the pair on three axes (concrete suggestion present, voice matches team norm, comment is actionable) and return a 0-3 score per axis. Aggregate the scores per row.
3. Sort by aggregate score; import the top 20% as a curated subset (Data → Sources → Import dataset; they land pending in the Review queue under Data → Generate → Details, and only accepted rows train) and put the best of them in the answer key on Data → Labels. Promotion is still an explicit user decision — the platform won't auto-promote teacher-scored rows on its own.

This compresses 8 hours of manual gold-curation into ~90 minutes of scripting + review. The trade-off: the teacher's score is a proxy, not a verdict. Spot-check 30 of the promoted rows manually before training; if more than 20% of those feel off-voice, tighten the teacher prompt and re-rank.

✓ Checkpoint: the Readiness overview ledger (Data → Sources → Details) should show its gold_set row at ≥ 100 rows (green) or 50-99 rows (amber, "100 recommended"). Below 50 it flags the answer key as a blocker. If the row is grey, nothing landed — open Data → Labels and check the Answer key card's entry count.

Splitting train, validation, test — by repo or author, not random

This is where the code-review recipe diverges sharpest from tutorial 1 and 2. Don't use a random split. Reviews by the same engineer are stylistically correlated; the same engineer's reviews appearing in both train and test inflates eval scores because the model can essentially "recognise" the reviewer's voice rather than generalising the critique reasoning.

Data → Split (the Train / Val / Test Split form, Run Split) cuts a random split off the ratios + seed — and its Disjoint By Field (optional) input is exactly the guarantee this section wants: give every row a key column, name it there, and every group lands whole in one split (the manifest's "Disjoint by" report shows per-split group counts and the ratio drift). Three useful keys for code-review:

• Split by repository. All rows from repo A go to train; all rows from repo B go to validation; all rows from repo C go to test. Best when your team works across many repos with overlapping author sets — the model has to generalise across codebases. Use when you have ≥ 5 source repos.
• Split by author. Engineer A's reviews go to train; engineer B's to validation; engineer C's to test. Best when one repo dominates your archive — the model has to generalise across reviewers. Use when you have ≥ 5 distinct reviewers contributing comments.
• Split by time. Reviews from Jan-Apr go to train; May to validation; June to test. The most realistic for measuring deployment readiness because production traffic is "later than training data". Disjoint-by won't order the months for you, so for this one pre-cut the files yourself and import the later months as an external benchmark (Results → Compare → External benchmarks). Use as a complementary check even when you've already done one of the above.

For a 400-row gold set, 80/10/10 produces 320 train / 40 val / 40 test. With a key-based split the per-split counts may be uneven — that's fine, the rule set handles small validation sets gracefully and the LLM-judge scoring works at any size. On every later re-split tick Keep the current N test examples so new rows never move the test examples into training.

Generate synthetic drills

The code-review recipe ships three synthetic playbooks. Each drills a different aspect of review-quality:

Paraphrase positives (POSITIVES_PARAPHRASE) — voice drill

Vary the wording of a review while keeping the underlying critique intact. "Use is None not == None" becomes "Prefer is None here — the loose-equality form coerces and is slower." Same critique theme, different phrasing. Goal: the model learns that the same diff can elicit many valid review wordings, all of which should hit the same theme. Generate ~80 rows from your gold set.

Hard negatives (HARD_NEGATIVES) — what NOT to emit

Two flavours for code-review specifically: (a) "LGTM" reviews paired with diffs that have real issues — the model needs to learn that empty-praise reviews are wrong; (b) reviews that MISS the actual bug while commenting on irrelevant style nits. Both train the model on the failure modes you want to avoid: silence when speaking up matters, and pedantry that misses the real concern. Generate ~50 rows.

Target a failure cluster (CLUSTER_TARGETED) — failure-mode patching

After a first training run + eval, the failure clusters on Results → Failures group failures into themes ("missed null checks", "wrong logger level", "off-voice"). This playbook seeds new training rows specifically targeting the weakest clusters — you point it at a cluster, it generates diffs that exercise the same pattern, and a teacher writes reviews. Skip on the first pass; run after the first eval to close the gap.

Open Data → Generate. Pick a Provider (Local (Ollama) is the free default; OpenAI / Anthropic give noticeably better paraphrases at code-review's voice — Save key to this project keeps the key) and Model Name, then open Details → Playbooks and run Paraphrase positives first with a target of 80. Generation runs as a background Job; the notification bell tracks progress.

Review the synth queue

Every generated row lands pending in the Review queue (Data → Generate → Details → Review queue, where you tick rows and use Accept selected / Reject selected). The Review queue under Data → Labels → Details groups rows by the playbook that produced them, with Accept all / Reject all per group, so you can review one category at a time. For code-review specifically, expect 30-50% soft-reject rate on the first pass — teacher models are stronger at code than at voice, so paraphrased reviews drift toward "ChatGPT voice" easily.

Per-row actions and reason tags:

• Accept — the row joins training on the next prepare-dataset run. For paraphrase rows: only accept if the new wording still sounds like something your team would say. "Sounds like a teacher model" is a reason to reject.
• Soft-reject with a reason tag:
  • not-our-voice — too formal, too verbose, too GPT-ey. The most common rejection on paraphrase.
  • misses-the-bug — review is grammatically fine but doesn't catch the real concern in the diff. Common on hard-negatives gone wrong.
  • too-pedantic — review flags something true but trivial; your team wouldn't bother commenting.
  • wrong-language — review references a syntax that doesn't apply to the diff's language (Python advice on JS code, etc.).
  • invented-issue — review describes a problem that isn't actually in the diff; the model would learn to hallucinate concerns.
• Purge this reason — group-level action that deletes all rejected rows tagged with one reason. Periodic cleanup; the platform never auto-purges.

The Synthetic quality section next to it (Data → Generate → Details) scores the generated rows. Low-scoring paraphrase rows almost always correlate with voice drift — review those first.

Training configuration

Open Train. It is one form — Base model (+ Browse), Task, Seeds (One run / Three seeds), Run name, a readiness line and the Train button (a confirm dialog shows the resolved config and an estimate, then Launch). The knobs below live under More options → Model and hyperparameters (epochs, batch size, learning rate) and LoRA and advanced training. The recipe defaults are sensible:

Base model

Qwen/Qwen2.5-Coder-1.5B-Instruct (type it in the Base model field or pick it via Browse). Pre-trained on code; instruction-tuned; good fit for the diff-to-comment shape. The 1.5B size fits comfortably on an 8 GB GPU. Alternative: Qwen/Qwen2.5-Coder-3B-Instruct for teams that want longer / more nuanced reviews and have a bigger GPU (12 GB+) — noticeably better at multi-issue diffs where the model needs to spot 2-3 issues in one diff. HuggingFaceTB/SmolLM2-1.7B-Instruct is the non-coder alternative if you don't want a code-pretrained base for some reason.

Adapter

LoRA, rank 16, alpha 32; leave target modules on auto (all linear layers for a model ≤ 2B). Standard for instruction-SFT.

Learning rate

2e-4. Same as the QA-shape recipes; LoRA's small parameter count tolerates the higher rate.

Epochs

3. For 300-500 row training sets, 3 epochs hits the plateau; 5+ overfits the voice. (An empty field lets the app size epochs to your row count.) The run's live-loss sparkline will plateau by step 200-300 if the dataset is healthy.

Batch size + gradient accumulation

Batch 2, accumulate 8 → effective batch 16. Diffs can be long (up to the 2-3k token context window); the smaller per-step batch keeps memory in check.

Expected runtime: 15-40 minutes on a single GPU (RTX 3060+ for the 1.5B, RTX 3090+ for the 3B), 60-90 minutes on CPU for the 1.5B (CPU is impractical for the 3B). Watch the run's live dashboard in the runs list under the form: if loss isn't dropping by step 100, Cancel the run and check your rows — the input data is probably misshapen (most common cause: diff column accidentally swapped with review column during mapping).

✓ Checkpoint: on Train, your run's row shows a live sparkline that drops from ~2.5-3.5 in the first few steps down to ~0.4-0.7 by the end. The bell shows the training Job with a percentage. When it completes, the automatic lift check scores the run and Results → Verdict shows the summary card — "Better than the base model" / "within noise" with the judge's correct / partial / wrong counts — plus Re-run lift check and Check across 3 seeds.

Read the trainability forecast

Before kicking off the training run, the platform pre-computes a trainability forecast (Train → More options → Plan and gaps, "Predicted gate-pass: ~N%"; the same number is the predicted_pass row of the goal ledger on Data → Sources → Details → Readiness overview). For code-review projects the forecast is noisier than for classification or QA — the LLM-judge gate has higher inherent variance than exact-match metrics because it depends on a teacher model's scoring of theme overlap, and theme overlap is fuzzier than token overlap.

For a healthy code-review project you want:

• Predicted pass probability ≥ 55%. Lower than the rag-protocol / classification thresholds because of the LLM-judge variance — a 55% prediction here is roughly equivalent to 70% on a classification project.
• Answer-key readiness ≥ 100% (i.e. ≥100 answer-key rows). Below this the forecast becomes too noisy to trust at all.
• Data ready = met (training rows + mapping + key-based split all green).

If the forecast is below 40%, training will likely fail the LLM-judge gate. Add more answer-key rows + run another paraphrase round — the Readiness overview ledger will tell you which component is weakest.

Evaluation: LLM-judge as the primary gate

After training, the automatic lift check scores the run and Results → Verdict shows the Pass/fail rules card (rule set, PASS/FAIL, the required rules that failed) above the judged summary. The judge is resolved from EVAL_JUDGE_BACKEND, else the project's eval_judge setting, else the first local model (Ollama) — never a cloud model over a reachable local one. For code-review projects the rule set covers three behaviours:

LLM-judge pass rate ≥ 0.70 (required)

The headline gate. A teacher model scores each candidate review against the reference review along three axes: (a) does the candidate identify the same critique theme as the reference, (b) is the suggestion actionable and concrete, (c) does the candidate's voice match the team norm. Pass = aggregate score crosses the threshold. This is the gate you actually care about.

F1 ≥ 0.40 (not required)

Token-level overlap between candidate and reference. Low because reviews paraphrase — two valid reviews of the same diff might share 30% of their tokens. Don't expect classification-style accuracy here. A model with F1 = 0.45 and LLM-judge pass = 0.78 is shipping-ready; a model with F1 = 0.75 and LLM-judge pass = 0.55 is over-memorising the gold and not generalising voice.

Safety pass rate ≥ 0.93 (not required)

Catches reviews that leak PII, suggest unsafe code, or use language inappropriate for your team's communication norms. Mostly defensive; should pass easily if your gold set is clean.

The scorecard under the Pass/fail rules card lists every rule with its value and threshold — "LLM-judge 0.72/≥0.70 passed, F1 0.43/≥0.40 passed, safety 0.96/≥0.93 passed." If LLM-judge passes but F1 is high (≥ 0.70), the model is likely over-memorising your gold; a too-perfect F1 on a generative task is a warning sign, not a success.

The failure clusters on Results → Failures group eval failures by theme. Common cluster headers for code-review:

• "missed the bug" — model commented on style while the diff had a real issue. The hard-negatives playbook targets exactly this.
• "off-voice" — model wrote a syntactically-fine review that doesn't sound like your team. Add more gold from your senior engineers' archive.
• "too verbose" — model wrote a 6-sentence essay when 2 sentences would do. Tighten gold to shorter exemplars; run paraphrase to compress.
• "invented issue" — model flagged a concern that isn't in the diff. The most worrying cluster — train on hard-negatives that include diffs with no real issues paired with appropriate-silence reviews.

When the eval fails

Common code-review-specific failure patterns and the fix for each:

Ship the model

Three deployment patterns for code-review models, from least to most assertive:

Draft suggestion mode (recommended first)

Deploy as a GitHub Action that posts each model-generated comment as a suggestion the human reviewer can accept-with-one-click or dismiss. This is GitHub's native review-suggestion mechanic (the syntax with ```suggestion blocks in PR comments). The model is offering input; the human is the decider. Zero false-positive cost — bad suggestions get dismissed and stay invisible to the PR author. This is where you should START.

Tee mode (review-assist)

Model posts comments visibly on the PR but tagged as [bot] with a "from code-review-bot — accept or dismiss" footer. Human reviewers see the suggestions alongside their own pass. Useful once draft-suggestion-mode acceptance rates climb above ~30% and the team trusts the model's signal.

Blocking review mode (after months of tuning)

Model's comments are posted as standard review comments equivalent to a human review. The PR is held until either the author addresses the comment or a human reviewer dismisses it. Don't do this until you have at least 3 months of draft-suggestion deployment data showing acceptance rate ≥ 50%. The cost of a bad blocking comment is high (slows the team); the cost of a missed dismissed suggestion is zero.

Export on Ship → Package → New export: pick the run, a Format (HuggingFace — the LoRA is merged into Qwen2.5-Coder at export, so the folder loads with plain transformers or vLLM; GGUF (CPU) for llama.cpp / Ollama; Docker, which bundles a serve.py and Dockerfile), quantization and deployment targets, then Run Export. The export appears in the Releases table and under data/projects/<id>/exports/; Ship → Deploy holds the serve / deploy plans for the targets you ticked, and Ship → Publish pushes a HuggingFace export to the Hub. Smoke-test first on Results → Try it (Provider = Trained run (this project)) with a few diffs. Latency: 1-3 s per diff on a single GPU.

Wire it as a GitHub Action that runs on PR open / synchronize. The Action pulls the diff via gh pr diff, posts it to whatever serves the export (a chat-completions call for vLLM / Ollama; wrap it in a thin /review endpoint), and writes the response as a draft-suggestion review comment via gh pr review --comment. Self-hosted runners with GPU access keep the round-trip private; the model never sees the internet, and your diffs never leave your network.

What's next

You have a deployed code-review nitpicker. Three obvious next moves:

Language-specific variants

One model handles all your languages OK; one model per language (Python, Go, TypeScript) handles each one noticeably better. Same recipe + workflow; segment your gold set by language and train per-segment. Routing happens at deploy time — the GitHub Action inspects the diff's file extensions and picks the right model.

Pull style-guide context via auto-RAG

Your team's style guide is a stable document. Upload it on Data → Sources and clean it; the passages become a retrieval index, and on Results → Try it the Answer from your documents toggle prepends retrieved passages to the diff at inference time (each reply shows its Sources). The model writes reviews informed by the exact style-guide rules — "according to section 3.2 of the team's Python guide, use pathlib not os.path for new code." Works without retraining the model; for the GitHub Action you prepend the same retrieved chunks yourself.

Active learning from acceptance signal

Every draft-suggestion the human reviewer accepts is a high-signal positive example. Every dismissed suggestion is a soft-negative. Capture both via webhook from your PR platform; add the accepted ones to the training rows and the answer key; retrain when the key grows by ~100 rows. Over a quarter, the model converges on the suggestions your team actually finds useful — a continuous-improvement loop powered by the review process itself.

Curious where to go after this? The tutorials hub covers other end-to-end recipes — span-extraction for structured field extraction, classification for binary security flags, rag-protocol for grounded-answer chatbots. Each one is the same workflow shape on a different recipe.