#Exercises: make the learning stick

No: generate() is seeded, so the same call yields byte-identical history every time. That determinism is the whole reason an exercise can predict a specific outcome and be right: a cache hit is provably identical, a detector fires on a knowable day, and the incident schedule is fixed. Change the seed and you get a different (but equally reproducible) history.

Because a real production log doesn't have it. There's no field that says "this answer was bad". If there were, monitoring would be trivial. The whole skill is inferring trouble from proxies (latency, refusals, a sampled judge). The schedule exists only as an answer key to grade our detectors against, exactly what you don't get in real life.

Much higher. The simulator gives most calls a moderate latency and a minority a long tail, so p95 sits well above p50. If p95 equalled p50 your latency would have no tail at all (every request nearly identical), which almost never happens in a real system calling a model over a network. Watching the gap is watching how bad the slow experiences are.

It drops, because cache hits cost $0 and would dilute the average. That's exactly why misses-only is the right denominator for "is a call getting more expensive?" you want the price of the work, undiluted by how often you skipped the work.

Not a bug. Cost per request is an extremely stable baseline metric, and its day-to-day spread is tiny, so even a modest absolute rise is an enormous number of standard deviations. A big z just means "this moved far relative to how much it normally wobbles." A noisy metric (like p95) needs a much bigger absolute change to reach the same z. That's the point of standardizing: it's comparable across metrics.

Novel-term rate. It fires the moment a question contains a word your baseline never saw ("iphone", "android"), which is instant and needs no model. Embedding drift is subtler (it can catch drift even in ordinary words) but moves more gradually. Cheap lexical checks are your early smoke alarm; embeddings are the deeper confirmation.

The model answers questions it can't answer well: here, refusing (or worse, confidently guessing) about a mobile app that isn't in the KB. Nothing errors, so every ops dashboard stays green while answer quality quietly rots. A failure that throws an exception gets caught in minutes; a silent quality failure can run for weeks until users leave.

Real. 0.615 + 0.089 = 0.704, still well below the 0.79 baseline, so the drop clears its own error bar. Compare a day at 0.685 ±0.077: 0.685 + 0.077 = 0.762, which does not clear the bar, so that day alone isn't yet conclusive. Reporting the margin is what stops you from declaring a regression on a wobble.

Because grading isn't free. A real LLM-as-judge costs a call per answer, and you have thousands a day. Sampling keeps the cost bounded. The trade: a smaller sample is cheaper but has a wider confidence interval, so it takes a bigger true drop to detect. More samples, tighter margin, higher cost: you pick the point.

persistence=1 catches the spike (one breach day is enough to fire); persistence=3 rides over it (a single day never reaches three consecutive breaches). That's the dial that separates a transient blip from a sustained regression. Neither is "correct": a one-day spike is worth a glance but not a 3am page, which is why you run a spike detector (persistence 1) and a trend detector (persistence 3) side by side.

Because normal p95 is already ~400ms, so the static threshold is below the healthy baseline, so it fires constantly. Ship it and you get alert fatigue: the team mutes it, and the muted alert is precisely the one that misses the real outage. A z-score against the actual baseline is "weird relative to normal," which no magic constant can be.

The largest cluster is the mobile app ("app", ~900 failures), the input-drift topic the KB can't answer, seen from the other side. And the vast majority of failures had no feedback at all (≈900 of ~1100). Thumbs are sparse, so you cannot wait for a thumbs-down to find failures. You mine proxies (refusals, terse answers) instead.

The mined failures become candidate eval cases in the exact JSONL shape the Evals dive uses. A human writes the gold answer, they drop into the regression suite, and now that failure can never silently return. Production → Observability → Evals → Production: the feedback flywheel.

SHAP/LIME explain a tabular model's prediction in terms of its input features, and an LLM has no fixed feature vector, just free text. LLM interpretability (probing, attention, mechanistic interp) is a research field, not a production practice, and attention weights are not feature attributions. The genuinely useful "why did it say that?" handles for an LLM app are its citations and the retrieved context (the RAG dive), not a borrowed tabular technique.

On the default history: all four incidents CAUGHT (with a few days' detection lag), and the latency regression detector "correctly silent": it didn't mistake the one-day spike for a trend. On --healthy: no alerts at all. Catching real incidents while staying quiet on a clean history is the whole game, and it's the same detector tuning in both runs.

You start getting alerts on the clean history, false alarms, because a 1.5σ bar is crossed by normal daily noise. That's the tradeoff made tangible: you bought faster detection on real incidents at the cost of paging on nothing. There's no threshold that gives you both; you choose where on the curve to sit.

The global detector stays silent. The enterprise slowdown, diluted across the 85% of traffic that's fine, keeps the overall p95 inside its normal noise band (it never reaches a persistent 3σ). The enterprise cohort's own p95 roughly triples and alerts loudly. Same metric, same detector; the only difference is what you grouped by. That gap is the whole point: an aggregate dashboard is necessary but not sufficient.

Two costs. First, noise: each extra split shrinks the sample per series, so small cohorts get noisy metrics that either miss real problems or cry wolf (which is why the example loosens persistence for per-segment detection). Second, sheer volume: the cross-product is thousands of series nobody can watch or alert on sanely. So you slice on the few dimensions that actually carry different risk, the ones where one group's experience really can diverge from another's, not every field available.