Observability and drift in production

A deployed model meets inputs you never trained on, and the world it learned keeps moving. Observability is how you notice when it stops being good — before your users do.

The RunEvent spine, recapped

From Track 3: every stage emits a RunEvent via emit_event() — a reason code from a lint-gated taxonomy plus a severity (info / warning / error / critical). One stream feeds every observability surface: the run-event timeline and failure clusters on Ship → Monitor, the audit explorer, and the support bundle. The emission is best-effort (wrapped in try/except at each call site), so a bug in observability can never break the actual data or training path. That single design choice is what makes the whole pipeline debuggable after the fact.

Drift: quality decays quietly

A model that scored 0.93 at launch can be 0.80 six months later without anyone touching it — because the inputs drifted. New slang, new product names, a new kind of customer. This is drift, and it's invisible unless you measure for it.

Drift detection: re-run the gold set on the live model

BrewSLM's Ship → Deploy stage (Track 3) has a drift check on every served version: it re-runs your gold set — the answer key under Data → Labels — against the deployed model and compares the pass rate to the baseline recorded at training time. You supply the live model's predictions — bring your own inference client and paste them into the check's form (the API and CLI can also call an endpoint URL themselves) — and run it on whatever cadence you choose; nothing schedules it for you. A significant drop emits deployment_drift_detected — the same gold set you built in Track 1, now a production sensor.

# drift check — Ship → Deploy → a served version → "Run drift check"
re-run the answer key vs the deployed model
  launch:  accuracy 0.93
  today:   accuracy 0.86   → drift!  emit deployment_drift_detected
# also: deployment_smoke_failed (promote-time), deployment_promote_blocked

When to retrain

Drift detection tells you that quality dropped; deciding what to do is judgment:

• Drift detected → gather recent real inputs, add them (via the review queue under Data → Generate → Details), re-split on Data → Split, re-train, re-evaluate against the same pass/fail rules on Results → Verdict. The Track 3 loop, triggered by a signal instead of a hunch.
• New failure clusters (Ship → Monitor, or Results → Failures for eval misses) → a targeted data fix (augment_from_cluster), not a full retrain.
• Stable → leave it alone. Retraining a healthy model just risks regressions.