The mental model of an SLM project

You now understand the parts. This lesson assembles them into the process you'll repeat for the rest of the course. Training a model is not a one-shot event; it's a loop you go around several times, and most of the work is not in the training step at all — it's in the data and the evaluation around it.

The loop

A fine-tuning project has six stages, and the last one usually sends you back to the start:

1. Data. Gather and prepare examples of your task: inputs paired with the outputs you want. Clean them, format them, split them. This is where most of your time goes and where most of your quality comes from.
2. Train. Run supervised fine-tuning (usually with LoRA) on a base model. This is the gradient-descent loop from Lesson 2, now over a Transformer.
3. Evaluate. Measure the trained model against held-out examples it never trained on, using a metric that fits the task. This tells you whether you actually improved.
4. Iterate. Look at where it fails, fix the data (add examples for weak cases, remove bad ones, rebalance), and retrain. Repeat until the metric clears your bar.
5. Export & deploy. Package the model into a servable artifact and stand it up behind an endpoint, with a version you can roll back.
6. Monitor. Watch real-world inputs for drift — the day the live data stops resembling your training data, quality slips and you loop back to step 1.

Your north star: the gold set

The most important artifact in the whole loop is the gold set — a curated batch of examples, with known-correct answers, that the model never trains on. It exists solely to measure quality. Every decision ("is this version better?", "did this data change help?", "is it good enough to ship?") is answered against the gold set. If you train on your evaluation data, your numbers become fiction — the model can memorize the test. Keeping the gold set separate and trustworthy is the discipline that makes all your metrics meaningful.

What "good" looks like at each stage

• Good data: representative of real inputs, correctly labelled, deduplicated, balanced across cases, with a clean train/validation/test split and a trustworthy gold set.
• A good training run: a loss that descends and stabilizes, no divergence, a checkpoint saved — and a model that does better on the gold set than the base did.
• Good evaluation: a metric that matches the task shape, reported honestly — including where the model still loses.
• A good deployment: a versioned, reproducible artifact you can roll back, with monitoring that will tell you when reality drifts.

How the rest of the Academy maps to this loop

Everything ahead is this loop, at increasing depth:

• Track 1 — SFT fundamentals: the why and what of each stage — datasets, tokenization, the training loop, LoRA, evaluation — in detail.
• Track 2 — Hands-on: you run the whole loop by hand in PyTorch and Transformers, with no platform, so nothing is magic.
• Track 3 — With BrewSLM: the same loop, with the platform automating the tedious parts (import, gates, eval packs, deployment) so you can iterate faster.
• Track 4 — Graduating: advanced moves that bend the quality/cost curve — distillation, warm starts, hyperparameter bake-offs, and proving your SLM is good enough versus a frontier model.

That completes the foundations. You can now talk about what a model is, how it learns, how language models produce text, the levers for steering them, and the shape of a real project. Next, Track 1 makes Supervised Fine-Tuning precise — starting with what SFT actually is, and when not to use it.