Internal knowledge-base QA via the qa-sft recipe — the simpler path

This is BrewSLM's canonical workflow for direct-answer QA over a small, stable knowledge base — the kind of corpus an internal-IT team at a 50-engineer company actually maintains. The classic shapes: "how do I get VPN access?", "where do I file expense reports?", "what's the office WiFi password?". These don't change weekly. The answers fit in the model's weights cheaply. You don't need a retrieval index to look them up — you need the model to remember them.

The end state is the model you'd actually deploy at a B2B engineering team: small enough to host on the same box as your Slack-bot worker, accurate enough that the helpdesk channel goes quiet, simple enough that one engineer can own the whole pipeline.

What you'll build

A direct-answer QA assistant. Concretely:

• "How do I get VPN access?" → "File a ticket in #it-help with the project you need access to, then run the Tailscale installer that IT sends back."
• "How do I file expenses?" → "Use Ramp. Photograph the receipt, tag the project code, and submit. Manager approval is automatic under $200."
• "How do I get on the office WiFi?" → "The network is brewslm-corp, password is in 1Password under Office WiFi. Mobile devices use the brewslm-guest network with a daily PSK posted in #office."

The model is a fine-tuned LoRA adapter on top of SmolLM2-135M-Instruct. Inference is ~200ms on a single GPU, 600-900ms on CPU (still fine for a Slack-bot). No citations in the output (the part rag-protocol does and qa-sft doesn't); when a fact changes, you update the gold set and retrain.

Why a small model (not regex, not a frontier API, not always RAG)

Three options people reach for when building an internal helpdesk bot. Use this comparison:

Regex dies the first time someone types "how the heck do I VPN" — the register your engineers actually use diverges hard from the canonical FAQ wording. Frontier-LLM works but you're paying per question and sending every employee's "what's our parental leave policy" out to a third-party API. The small fine-tuned model is the middle path: handles paraphrase, stays on your hardware, costs nothing per query at runtime.

The remaining decision is qa-sft vs rag-protocol — both are BrewSLM recipes for QA, and they look almost identical from the outside. The recipe-choice section below is the real load-bearing piece of this tutorial: the answer changes based on properties of your corpus, not preferences. Read it carefully before you start building.

Choose your dataset

You need (question, answer) pairs. The answer is the canonical thing the model should say when asked the question. Three common starting points:

Your own knowledge base

Most internal-IT teams already have the data, scattered across three places. Confluence / Notion pages where an H2 is the question and the paragraph below is the answer. Slack threads where someone asked, IT replied, and the reply got a thumbs-up. Resolved tickets in Jira / Linear with the "answer" field populated. Extract Q→A pairs from all three; you'll find 40-80 high-signal ones in any half-mature engineering org.

Public starting points (for warmup)

SQuAD ships 100k QA pairs over Wikipedia paragraphs — the canonical QA benchmark. Natural Questions is Google's dataset of real user questions answered from Wikipedia. Either gets you to a working pipeline in 30 minutes; swap in your real corpus once you've verified the platform end-to-end.

Resolved helpdesk archives

A year of resolved internal-IT tickets already contains canonical Q&A pairs — the question is the ticket title, the answer is the first reply that closed the ticket. Filter for tickets that resolved without follow-up and you've got pre-validated gold candidates.

Ingest and map

In BrewSLM, create a new project: on the projects home click + New Project, give it a name and a brief ("Answer internal IT helpdesk questions from our knowledge base, directly, no citations"), then Create project. On the project's Overview check the task; if it isn't qa-sft, open Train → Task → Change and pick it. The recipe pre-fills the adapter (qa-pair), the task profile (instruction_sft), the scoring mode (field_match), and the pass/fail rule scaffold (exact_match + F1 + LLM-judge).

Open Data → Sources → Import dataset (auto-mapping) → Import dataset →. Your CSV or JSONL should look like:

question,expected,rationale
"How do I get VPN access?","File a ticket in #it-help with the project you need access to, then run the Tailscale installer that IT sends back.","Standard onboarding flow; the project-scoping prevents broad Tailscale acls."
"What's the office WiFi password?","The network is brewslm-corp, password is in 1Password under Office WiFi. Mobile devices use brewslm-guest with a daily PSK posted in #office.",""
"How do I expense a $30 dinner?","Use Ramp. Photograph the receipt, tag the project code, submit. Under $200 manager approval is automatic.",""

Two required columns (question, expected) and one optional (rationale). Rationale is for your future-self — some eval handlers use it; the qa-sft recipe doesn't require it for training. The import wizard scans your columns and proposes a binding; confirm it when the preview rows look right.

✓ Checkpoint: Data → Sources → Details → Readiness overview now shows your imported row count (e.g. "62 trainable rows"), and Sources summary in the same drawer lists your imported CSV/JSONL. If the mapping is wrong, the preview will show empty or mis-shaped rows — check Data → Split → Details → Schema mapping and re-run the import with the correct column for each field.

Cleanup

Open Data → Prepare → Details → Quality and safety. Three deterministic scans worth running:

• Strip Slack / Confluence markup. Slack data carries <@U02ABC> user mentions, channel refs like <#C03XYZ|it-help>, and emoji shortcodes (:tada:). Confluence exports carry <p>, <strong>, and the occasional rogue <ac:structured-macro>. Strip them; the model doesn't need to learn that <@U02ABC> means "IT manager".
• Dedup near-identical questions. People ask "what's the WiFi password?" five different ways. The platform's dedup signal in the same panel clusters them. The right move is NOT to drop them all. Keep ONE in the answer key with the canonical answer, and let the paraphrase synth playbook regenerate the other phrasings later as training rows — question variation in training, one canonical answer per question.
• PII signals. Internal helpdesk archives often have employee email addresses and Slack handles in the answer text. Decide per row — "open a ticket at helpdesk@brewslm.com" stays; "ping jane@brewslm.com directly" probably drops (Jane will leave the company and the model will keep recommending a dead address).

You don't have to clean everything before training; you do have to clean the rows you promote to the answer key.

Pick the recipe: qa-sft or rag-protocol?

This is the load-bearing decision in the whole tutorial. BrewSLM ships two recipes for question-answering and they look surprisingly similar from the outside. The difference is which property of your data they exploit:

• qa-sft memorises the facts into the model's weights. You ask, it remembers, it answers. When facts change, you retrain.
• rag-protocol trains the model to use a retrieval index — given a chunk of context, cite the source, refuse if context is insufficient. The facts live in the index (BM25 over your corpus), not the weights. When facts change, you re-index; no retraining needed.

Use this decision matrix:

The internal-IT helpdesk inside a 50-engineer company is the canonical qa-sft case. Facts are stable across quarters. The corpus is small — 40-80 canonical Q&As cover 95% of helpdesk volume. Citations aren't required because the answer is itself the source-of-truth from the team that maintains the policy.

An ecommerce FAQ that updates monthly, a legal QA system that touches new statutes, or a customer-support bot covering 600 product SKUs — those are rag-protocol territory. Tutorial 1 walks through that path.

Domain packs (optional)

BrewSLM ships a few domain packs (support, legal, ecommerce) that bundle cleaning rules, eval thresholds, and synth-recipe constraints scoped to one vertical. The support pack is the closest match to internal-IT helpdesk — it pre-loads PII patterns common to ticket data.

For most internal-KB use cases domain packs are optional. The platform defaults work fine; the support pack is a refinement you can apply later from Settings → Tools → Domain Packs if you find yourself doing the same PII cleanup on every project. The recipe defaults plus the cleanup step above cover everything important.

Build the answer key

The answer key — the gold set — is the trusted reference your model is evaluated against, under Data → Labels. For qa-sft this is the most important data work you'll do — every answer-key row is a fact the model commits to memory. Aim for 40-80 rows; quality matters more than quantity.

Path A — manual seeding from your canonical answers

Open Data → Labels. Confirm "What kind of task is this?" if it asks, then work in the Answer key card (the Answer-key workbench with coverage and suggested rows is in the step's Details drawer). For each row you add:

1. Write the question the way a real teammate would ask it. If you have Slack logs, copy the actual wording — "how do I get on VPN" beats the textbook "How do I request VPN access?".
2. Write the expected answer. Make it short, factual, and complete in a single response. If the canonical answer is currently a 12-sentence Confluence page, distill it to the 2-3 sentence version that would fit in a Slack reply.
3. Optionally fill rationale — a one-line note on why this answer is right. Helps your future self when you're auditing the answer key six months later.

Spend an hour here. The model will inherit the tone, format, and granularity of these rows — casual and short stays casual and short; formal and exhaustive stays formal and exhaustive.

Path B — LLM-assisted promotion from raw exports

If you bulk-imported a few hundred Confluence pages or resolved tickets, use the same "promote from raw" flow other tutorials use:

1. Bulk-import the raw pages / tickets as candidate rows.
2. Run a teacher model (Ollama is the free default; OpenAI / Anthropic / DeepSeek work if you have keys) against them from Data → Generate — Provider, Model Name, Load from Cleaned Data — with the generation mode set to Single-turn Q&A, so it extracts (question, answer) pairs.
3. Every extracted pair lands in the review queue (Data → Generate → Details → Review queue) with review_status="pending". Accept the good ones; they're promoted to the answer key.

This compresses 3 hours of manual gold-writing into 30 minutes of review. The platform will not auto-accept LLM-generated gold — each promotion is an explicit click.

✓ Checkpoint: the goal ledger's answer-key readiness row (Data → Sources → Details → Readiness overview) should now be green (60 rows ready) or amber (fewer than the recommended 50). Amber is fine for now — the paraphrase playbook in the next section will multiply your seed. If it says nothing has been promoted, open Data → Labels and confirm rows landed in the Answer key card.

Splitting train, validation, test

BrewSLM splits when you click Run Split on Data → Split (the Train / Val / Test Split form; the split adapter reads qa-pair). The default ratios are 80/10/10 with a deterministic seed. For qa-sft, uniform random off the canonical config is fine.

This is genuinely different from the other tutorials. Span-extraction cares about template-leakage; code-review cares about author-leakage. qa-sft doesn't have an equivalent leakage trap — questions are independent, answers paraphrase-stable. Random split is honest here.

Override the default ratios in the same form only when your answer key is small (under 50 rows): use 70/15/15 so val/test have at least 7-10 rows each. Below that the eval metrics get too noisy to read.

Generate synthetic drills (just paraphrase)

The qa-sft recipe ships two synth playbooks, and the headline one for a fresh project is the paraphrase playbook. This is intentional — qa-sft is the smaller, simpler recipe and it deliberately doesn't ship the modes that classification or span-extraction do. If you've read Tutorial 2 (SQL injection) or Tutorial 3 (invoice extraction), you'll have seen hard-negative and class-balance-fill playbooks; those are not available for qa-sft, and that's by design. There are no "negative classes" in direct-answer QA the way there are in classification — every question has a correct answer; there's nothing to be a "negative" against.

qa_sft_paraphrase — the workhorse (POSITIVES_PARAPHRASE mode)

For each gold row, generate N paraphrases of the QUESTION while keeping the ANSWER verbatim. Goal: the model learns that "how do I get VPN?", "I need VPN access, where do I go?", "VPN setup steps?" all map to the same canonical answer. Generate ~3-5 paraphrases per gold row; a 60-row seed expands to 240-360 trainable rows.

qa_sft_cluster_targeted — for iteration 2 (CLUSTER_TARGETED mode)

After your first training pass + eval, the platform's failure-cluster surface will identify question-types the model is weak on. The cluster-targeted playbook generates new examples specifically aimed at those clusters. Don't run this on iteration 1 — there's no failure cluster yet. It's the right tool for round 2.

Open Data → Generate → Details → Playbooks. The qa-sft recipe surfaces the paraphrase card; click it, set target count to 3-5× your answer-key size, pick a backend. Generation runs as a background Job — the notification bell tracks progress, and Details → Playbook center keeps the run history.

Review the synth queue

Every paraphrase row lands in the Review queue (Data → Generate → Details → Review queue) with review_status="pending". The queue groups rows by source answer-key row, so you can scan "here are 5 paraphrases of the VPN question" in one screen. Per-row actions:

• Accept — the row joins the training corpus on the next split.
• Reject (soft) — marked with an optional reason tag (answer-paraphrased, question-meaning-drifted, too-similar-to-seed). The row stays on disk for audit.
• Purge — a section-level action that physically deletes all rejected rows (optionally filtered by reason).

The key check for qa-sft paraphrase rows is "does the answer match the seed verbatim?". If yes (most rows), accept. If the teacher paraphrased the answer too — even slightly — soft-reject with the answer-paraphrased reason. You want question variation, not answer variation.

Expect to accept ~70-85% on first pass. If you're rejecting more than half, the teacher prompt is drifting and the playbook card's prompt editor lets you tweak the system message.

Training configuration

Open Train. The form shows Base model, Task, Seeds and Run name; the values below live under More options → Model and hyperparameters and LoRA and advanced training. The qa-sft recipe defaults are sensible — for a first run, accept all of them:

Base model

HuggingFaceTB/SmolLM2-135M-Instruct. Small, instruction-tuned, runs on consumer hardware. Alternatives the recipe surfaces: Qwen/Qwen2.5-0.5B-Instruct (slightly better quality at ~4x the size), Qwen/Qwen2.5-3B-Instruct (noticeably better, needs more VRAM).

Adapter

LoRA, rank 16, alpha 32, target modules q_proj,k_proj,v_proj,o_proj. Standard. No separate classification head, no custom output projection — qa-sft uses the base model's standard LM head; the adapter just trains its attention projections.

Learning rate

2e-4. Higher than full fine-tuning because LoRA has fewer trainable parameters.

Epochs

2. Slightly fewer than the classification or span-extraction tutorials because the task is simpler — the model is memorising direct answers, not learning a discriminative boundary. For 250-row training sets, 2 epochs typically hits the loss plateau; 4+ overfits and the model starts emitting verbatim training rows even on questions it shouldn't.

Batch size + gradient accumulation

Batch 4, accumulate 4 → effective batch 16. Adjust down on GPUs under 8 GB.

Expected runtime: 3-10 minutes on a single GPU (RTX 3060 or better), 10-25 minutes on CPU. The runs list below the form shows live loss; if loss isn't dropping after the first 50 steps, cancel the run and check your data — the dataset is probably misshapen.

✓ Checkpoint: in the runs list under Train, your run's loss drops from ~2-3 in the first few steps down to ~0.3-0.6 by the end. The bell shows a "training" notification with a percentage. When complete, BrewSLM runs the lift check against the base model on its own; Results → Verdict shows the summary card (or an Evaluate this run button if the check didn't run).

Read the trainability forecast

Before kicking off the training run, the platform pre-computes a trainability forecast: given your current data + answer key + base model, what's the predicted pass probability? It lives under Train → More options → Plan and gaps; the goal ledger (Data → Sources → Details → Readiness overview) shows the same number as the predicted_pass row.

qa-sft's pass probability tends to be higher than rag-protocol's for the same-size data set. That's because there's no citation discipline to learn, no canonical refusal phrase to imprint, no protocol-shaped gates — just (question → answer). Healthy targets for a fresh internal-KB project:

• Predicted pass probability ≥ 70%. qa-sft on 60+ answer-key rows + paraphrase synth typically clears this. Below 50% usually means your answer key is genuinely too thin or has inconsistent canonical answers (one question with multiple slightly-different golds — see the cleanup section).
• Answer-key readiness ≥ 100% (≥ 50 rows for qa-sft's default minimum). Below that the forecast becomes noisy.
• Data ready = met (training rows + mapping + splits all green).

If the forecast is below 50%, add data before you train. The goal ledger's blockers panel tells you which component is weakest.

Evaluation: exact_match + F1 + LLM-judge

After training, the platform automatically evaluates against the project's rule set; Results → Verdict shows the Pass/fail rules card (PASS / FAIL and the failed required rules) below the summary card, both naming the run they describe. For qa-sft projects the scaffolded rule set carries four rules:

Exact match ≥ 0.45 (required)

Share of predictions where the model emitted the gold answer verbatim. 0.45 is intentionally not strict — exact_match is a coarse signal for natural-language answers because "Visit Settings → Security" and "Go to Settings, then Security" both say the same thing but differ on every token. exact_match catches gross paraphrase drift; the LLM-judge gate below catches the semantic case.

F1 ≥ 0.60 (required)

Token-overlap F1 between prediction and gold. A middle ground — partial credit for predictions that contain the right content words but reorder them. Standard SQuAD-style F1.

LLM-judge pass rate ≥ 0.75 (required)

A teacher model reads (gold, prediction) and scores whether they say the same thing semantically. This is the gate that actually matters for natural-language QA. It catches the "Visit Settings → Security" / "Go to Settings, then Security" case as a pass even though exact_match fails. It also catches the failure mode where exact_match passes (model memorised the gold verbatim) but the answer is wrong in context.

Safety pass rate ≥ 0.93 (not required)

Refusal / off-topic / adversarial input handling. Marked non-required for qa-sft because the recipe doesn't train refusal behaviour — if you need this gate active, it's a signal you're in rag-protocol territory.

The Pass/fail rules card names the failed required rules, and the goal ledger's eval_pass_rate row expands into the per-rule breakdown so you see exactly which rule failed — "exact_match 0.41 / ≥ 0.45 failed, f1 0.63 / ≥ 0.60 passed, llm_judge 0.78 / ≥ 0.75 passed". The exact_match-vs-LLM-judge split is the signal you'll learn to read.

When the eval fails — and the reroute escape hatch

Common qa-sft failure patterns and the fix for each:

The reroute-to-RAG flow (when memorisation stops being enough)

BrewSLM's post-eval decision engine looks at your eval result and the shape of your gold set. When it sees retrieval-shaped signals — low pass rate combined with high answer-diversity in your gold (many distinct factual answers) — it surfaces a recommendation card on Results → Failures (and as a Coach notice at the top of Results): "You're memorising what should be retrieved. Reroute to rag-protocol?"

Expand the "Why this fired?" disclosure. You'll see the actual numbers ("answer-diversity ratio 0.82 ≥ 0.70 threshold", "exact_match 0.31 < 0.45 gate"). If the trace matches your read of the data, click "Switch to RAG (keeps your answer key)". The platform runs clone_project_for_rag, which does three concrete things:

1. Creates a sibling project with your project name + " (RAG)" suffix. The new project's parent_project_id points back at the qa-sft project.
2. Forces runtime_config.rag_first=True and auto_rag.enabled=True on the clone. At inference time the playground uses the BASE model (no LoRA) plus BM25 retrieval — no new training step.
3. Carries your answer key + prepared splits forward and builds the BM25 index immediately, so the sibling's playground works the moment the clone completes.

The original qa-sft project stays put — nothing is destroyed. You now have two projects side-by-side. Open Results → Try it on each, ask the same 10 questions, compare. If you accept the reroute, the rest of the workflow is Tutorial 1's — answer key, cleanup, and deployment pattern all carry forward; what changes is the recipe and the rule set.

Ship the model

Once the rule set passes, ship in three steps:

1. Export the LoRA adapter. Open Ship → Package: in the New export form pick the run, the format (Hugging Face), the deployment targets, then Run Export; the export shows up in the Releases table. The platform writes the adapter weights, the tokenizer config, and a deploy manifest into data/projects/<id>/exports/. The adapter is ~5-15 MB; the base model is loaded fresh at deploy time.
2. Deploy via vLLM (or Ollama). The recipe's target_profile is vllm_server by default. The export bundle includes a vLLM launch script:

   cd data/projects/<id>/exports/run-2026-06-05
   ./deploy-vllm.sh
   # Serves the base model + LoRA adapter on localhost:8000
   # Exposes the OpenAI-compatible /v1/chat/completions endpoint

   Ollama variant: ./deploy-ollama.sh. Either gives you a standard chat-completions endpoint on a local port.
3. Wrap in your own /ask microservice. BrewSLM doesn't expose a hosted /ask endpoint — you wrap the chat-completions endpoint in a thin HTTP service that handles the request shaping (your system prompt, your auth, your rate limiting). For a Slack-bot use case the wrapper is ~30 lines of Python: receive a Slack event, call /v1/chat/completions with the user's question, post the response back to the channel. Same deploy pattern other tutorials use; nothing qa-sft-specific.

What's next

You have a deployed direct-answer internal-KB assistant. Three obvious next moves:

Refresh every N weeks when the KB changes

qa-sft memorises facts. When facts change, you retrain. Set a recurring calendar item — every 4-8 weeks for an internal-IT helpdesk. Update the answer key with new canonical answers, re-run the paraphrase playbook on the new rows, retrain. < 30 minutes once you've done it once.

Grow the corpus thoughtfully (don't just dump everything in)

The temptation when the model works is to throw every Confluence page at it. Resist. Each added row is a fact the model has to keep memorised; pile on too many and the memorisation pressure costs accuracy on the original rows. Add new answer-key rows when you have a real "we got this question N times and didn't have a good answer" signal.

Watch the reroute signal — the honest "you've outgrown qa-sft" tripwire

When your corpus passes ~200 facts OR your KB updates more than once a month, the post-eval decision engine usually fires the reroute-to-RAG recommendation. Accept it when it fires. qa-sft was the right tool when the corpus was small and stable; rag-protocol is the right tool when it isn't. Walk through Tutorial 1 from that point — your answer key is already carried forward.

The next tutorial in this series picks a different recipe shape: invoice field extraction with the span-extraction recipe — structured output with character offsets, JSON-schema gold sets, and the span-set eval handler. Different shape, same end-to-end workflow.