Small language model roadmap for ML engineers

New ML engineers

You know Python, but not yet the mechanics of tokens, attention, loss, or why a base model behaves the way it does.

• Do Foundations first
• Then move into SFT fundamentals

Experienced LLM engineers

You already prompt and ship models, but want a tighter grasp on LoRA, dataset design, gold sets, and the trade-offs between SFT, RAG, CPT, and DPO.

• Start with what SFT changes
• Then choose the right objective

Teams building domain models

You already know the theory and want the shortest path from custom data to a local, auditable workflow with repeatable evaluation.

• Run the hands-on track
• Then do the BrewSLM platform track

How models work

• What is a model?
• Next-token prediction

How to choose the adaptation path

• Pretraining vs fine-tuning vs RAG
• When continued pretraining is the right answer

How to choose the starting checkpoint

• Base vs instruct models
• Picking a base model

1. Foundations

Build the vocabulary and mental models behind small language models, tokens, attention, base vs instruct, and the SLM trade-offs.

2. SFT fundamentals

Learn supervised fine-tuning, loss masks, chat templates, dataset quality, LoRA, GPU memory, evaluation, and when not to fine-tune.

3. Hands-on

Load a base model, build a custom dataset, train with LoRA or QLoRA, evaluate honestly, and ship a runnable artifact in Python.

4. With BrewSLM

Move from script-level understanding into a platform workflow for ingestion, preflight, training jobs, eval gates, export, and deployment.

5. Advanced

Go beyond one SFT run with distillation, DPO/ORPO, quantization, multi-task training, serving, drift detection, and production feedback loops.

Use SFT when

You need the model to respond in a better format, tone, task shape, or narrow behavior on data you can label clearly.

• What is SFT?

Use RAG when

You need fresh or private facts at inference time and the problem is retrieval, not permanently changing the model's behavior.

• Pretraining vs fine-tuning vs RAG

Use continued pretraining when

The base model does not speak the language of your domain well enough at the token level and SFT alone cannot close the gap.

• Continued pretraining

Use DPO or ORPO when

You already have reasonable outputs and need the model to prefer better responses over worse ones using chosen/rejected data.

• Preference tuning: DPO and ORPO

Start learning

Open the Academy overview if you want the whole curriculum with progress tracking and per-track lesson lists.

Start building

If you already know enough theory, jump back to the product and run the workflow on your own custom data.

Keep a reference nearby

The glossary is useful when you want a quick definition without leaving the learning path.