Decoding controls: temperature, top-p, stop tokens

The same trained model can produce wildly different outputs on the same prompt depending on how you decode. Decoding is the bridge from "a probability distribution over the next token" to "actual text" — and it is governed by knobs that have nothing to do with training. Every developer eventually mistakes a decoding problem for a training problem, retrains, and discovers nothing changed. This lesson is so you skip that step.

Decoding is not training

The model produces a probability for every token in the vocabulary at every position (Lesson 0.6). Decoding is the rule that picks a token from that distribution. The training process determines the distribution; the decoding rule determines which token in it you actually emit. Tuning decoding does not change a single weight — it changes which path you take through the model's predictions.

Greedy vs sampling

Two families of decoding rule. Greedy always takes the most likely token. It is deterministic — same prompt, same output — and tends toward the average; great for evaluation, can be repetitive for generation. Sampling draws a token from the distribution. It is non-deterministic, adds variety, and is the right default for chat / writing tasks where blandness is itself a failure.

In the HF Transformers API: do_sample=False is greedy; do_sample=True is sampling. For classification or structured extraction you almost always want greedy. For chat / FAQ-style generation you almost always want sampling.

Temperature: sharpen or flatten the distribution

Temperature (T) divides the logits before softmax. T = 1 is the trained distribution. T < 1 sharpens — high-probability tokens become more likely, low-probability tokens become less. T > 1 flattens — more variety, more risk. Extremes are easy to reason about: T → 0 approaches greedy; T → ∞ approaches uniform random.

• Classification / extraction: T = 0 (or greedy). Deterministic, no surprises.
• FAQ / structured chat: T ≈ 0.2–0.4. Slight variety; still anchored to high-confidence tokens.
• Creative / open-ended: T ≈ 0.7–1.0. The model takes risks.
• > 1.2: rarely useful. Output is plausible-looking gibberish more often than it's interesting.

Top-p (nucleus) and top-k: chop off the tail

Temperature reshapes the distribution but still allows any token. Top-p and top-k truncate the tail before sampling, so the model never picks an obviously bad token.

• Top-k: keep only the k most likely tokens, renormalise, sample. Simple, effective. k ≈ 40 is common.
• Top-p (nucleus sampling): keep the smallest set of tokens whose cumulative probability ≥ p, then sample. p ≈ 0.9 is a sane default. Adapts to the distribution's shape (peaky → few tokens; flat → many).

The standard combo for chat is temperature=0.7, top_p=0.9, top_k=0 (top-k off, nucleus on). For deterministic tasks all of these are off (greedy).

Stop tokens and max_new_tokens: don't let it ramble

Stop tokens tell the model to halt generation when it produces a particular token (often the chat template's end-of-turn marker, e.g. <|im_end|>). Without one — or with the wrong one — the model keeps going past your answer and emits the next user turn it imagines, or just trails off. Instruct models bake their end tokens into the tokenizer; chat templates set them automatically.

max_new_tokens is a hard cap. Always set it. A chat answer with a forgotten stop token and no cap will fill its context window with hallucinated conversation. For a classification task that emits one label, max_new_tokens = 4 is fine; for a chat answer, 256–512.

Repetition penalty: small fix for a common bug

Sometimes a model loops: it generates a phrase, then re-generates it, then re-generates it. The repetition penalty reduces the probability of any token that has already appeared in the output. A small value (1.05–1.15) usually breaks the loop with no quality cost. Don't go above ~1.3 — you'll start suppressing legitimately repeated words like "the."

Time to first token (TTFT) and streaming

Decoding settings affect throughput (tokens per second) but the user-visible latency starts before any token is produced: time to first token (TTFT) is the gap between sending the prompt and seeing the first character of reply. For a chat UX, TTFT under ~200 ms feels instant; over ~1 s feels broken. Streaming — emitting tokens as they're produced rather than buffering the whole answer — masks the rest of the latency by giving the user something to read immediately. Most serving stacks support it; turn it on for any interactive use.

Decoding lives at inference time, but it interacts with everything you've trained. Track 2 will pick the actual values in code; Lesson 1.20 covers another pre-training-time concern: the dataset format you've been handed.