Bonus dive

#Chapter 16: The Next Six Weeks

This is the textbook chapter for the Observability deep dive, a bonus dive that pairs with Production and Evals. The README is the lab manual; this is the lecture. It covers the failures that never throw an exception, the handles you actually have when your system's core component takes free text and returns free text, and the craft of alerting that catches incidents without training everyone to ignore it.

#16.1 Nothing turned red

Here is a failure story that has happened, in some variation, to nearly every team running a language model application, and will happen to yours.

The launch went well. The eval suite passed, the demo impressed, the dashboards came up green. Weeks pass. Then a slow realization assembles itself from fragments: support tickets sound frustrated, a customer mentions offhand that "the assistant got worse," someone finally reads a sample of recent transcripts and winces. The investigation finds the cause was weeks old. Maybe users had gradually started asking about a product area the knowledge base never covered, and the model had been gamely answering badly the whole time. Maybe the provider had swapped the model version underneath, and answers got quietly terser. Nobody was paged, because nothing failed. No exception, no error rate, no red. The system did exactly what it was built to do, fluently, while its quality rotted.

Traditional software mostly fails loudly: exceptions, timeouts, 500s, all of them countable, all of them wired to pagers decades ago. Language model applications add a class of failure that is silent by construction, because the system's whole job is producing plausible text, and it will keep doing so at exactly the same latency, cost, and error rate while being wrong more often. Your existing dashboards cannot see it. That is not a gap in your dashboards; it is a property of the workload, and it needs its own instruments.

Hence the one big idea, positioned deliberately against the rest of this series:

A prototype is judged once. A production system is judged continuously, so your quality has to be a trend you watch, not a number you checked at launch.

The Evals dive (Chapter 5) told you a change was better today, on the questions you had today. Production (Chapter 8) told you what one request did. This dive is about the six weeks after, and it completes a triangle those two chapters left open.

#16.2 Learning detection honestly: the simulator

This dive needed history to teach on (you cannot learn to spot a trend from one request), so its equivalent of Production's mock model is a deterministic traffic simulator: forty-two days of realistic request logs for the same support assistant, with four real incidents buried inside. Users drift toward asking about an unsupported mobile app. A silent model swap makes answers terser for a stretch, then rolls back. A prompt change bloats the context and cost creeps. And one day, latency spikes and recovers, a deliberate red herring to teach the difference between a blip and a regression.

Two design decisions in that simulator carry more pedagogy than anything else in the repo, and both are principles this series' AUTHORING-LESSONS.md formalized. First: the log records contain only what a real system would observe (question, latency, cost, tokens, whether it refused) and no "was this answer good?" field. The ground-truth incident schedule exists separately, as a private answer key used only to grade your detectors at the end. Leak the label into the data and every detector becomes a fake that reads the answer instead of inferring it. Second: the incidents are injected by genuinely changing the simulated behavior, never by stamping numbers into a chart. The quality regression makes the answers actually terser and more evasive, so a judge scores them lower because they are worse; the cost creep adds real prompt tokens. A detector that works here works on reality, because the signal falls out of behavior, not out of a rigged flag.

Sit with the absence of that quality field for a moment, because it is the entire problem statement. In classic machine learning operations, labels eventually arrive (the loan defaulted or did not) and you can score yesterday's predictions. In an LLM application, almost no request ever gets a verdict. Everything that follows is a strategy for inferring health from what you do have.

#16.3 Numbers need context: metrics and baselines

The first handles are the operational ones, cheap because they are already in your logs (the exact trace shape Production emits, which is the point of the pairing): request volume, latency, cost per request, error and refusal rates, cache hit rate.

Two habits turn those from data into instruments. The first is an old operations lesson: watch percentiles, not averages. The p50 tells you the typical experience; the p95 tells you what your unluckiest users feel, and the tail is where trouble shows first while the average smiles blandly over it. The second is this dive's methodological spine: a number means nothing alone. "Cost per request is $0.00006" is noise; "up from a baseline of $0.00003, a shift many standard deviations beyond normal variation" is an incident. The lab's machinery learns what normal looked like from a clean baseline window, then scores every new day as a z-score, how many baseline standard deviations from usual. The elegance is that z-scores are unitless: the same "about three sigma is weird" intuition works for latency, cost, and refusal rate alike, with no hand-tuned threshold per metric. You will meet this trick everywhere real monitoring is done, because it converts every metric into the same question: how surprising is today?

#16.4 The two drifts

The silent failures come in two species, and they need different detectors; a single "quality number" would conflate exactly the things you need to tell apart.

Input drift is the world changing: users start asking things your system was never built for. Nothing errors, because answering badly is not an error. The lab detects it three ways, deliberately cheapest first. Novel-term rate is pure string counting: what fraction of today's questions use words the baseline never saw? (When a company launches a mobile app and the assistant's corpus knows nothing about it, "iPhone" and "notification" light this up within days.) Embedding drift asks the same question in meaning-space, using Chapter 4's machinery for a new purpose: how far has today's center of mass moved from the baseline's? And PSI, the population stability index, is the classic distribution-shift statistic from credit-risk modeling, shown here on a numeric feature so you have seen the real thing in its native habitat. The layering is the lesson: reach for string counting before embeddings, and embeddings before statistics, because the cheap detector you actually run beats the sophisticated one you meant to set up.

Quality drift is your system changing: same questions, worse answers, classically because the provider updated the model underneath you (a thing that genuinely happens, and which pinned model versions only partly protect against). Latency, cost, and errors never move. The only instrument that sees it is judgment, and judgment costs money, so you sample: a slice of each day's answers, scored by a judge (Chapter 5's LLM-as-judge, or the lab's deterministic rule-based stand-in). And because a mean over twenty sampled answers is a point estimate, the lab reports it with a confidence interval and enforces the discipline that follows: a dip inside the error bars is not a regression yet. Chapter 5's statistics were not academic; here they are the difference between paging someone at 2 a.m. and waiting for evidence.

The lab has you watch both detectors run against the same history: the sampled quality score falls clear of its error bars during the model swap and recovers at rollback, while refusal rate does its own unrelated dance during the input drift. Two diseases, two instruments, separable diagnoses. That is what "the metric that moved names the stage that broke" (a lesson from this series' own capstone postmortems) looks like as a monitoring practice.

#16.5 Alerting: the craft of being told

A dashboard nobody watches is a decoration; the point of all this is to be told. And alerting is where monitoring becomes a human-factors discipline, because the failure mode is not technical, it is psychological, and it has a documented body count in other fields: hospital alarm systems fire so often that clinicians tune them out, and the phenomenon (alarm fatigue) has contributed to real patient deaths. Operations teams are the same animal. The naive alert (page if p95 > 300ms) fires on ordinary bad Tuesdays until someone mutes it, and the muted alert is the one that sleeps through the real outage. Every alert that wastes an engineer's trust withdraws from an account that funds the response to the one that matters.

The lab's detector is built from three parts that together encode the craft. A baseline z-score (how surprising is today, in units of normal variation). A direction (up is bad for latency; down is bad for a quality score; a metric improving should never page). And persistence: require the breach to hold for N consecutive days before paging. Persistence is the dial that separates the simulator's one-day latency spike (which should not page as a regression, and does not) from the multi-week drifts (which should, and do).

And here the dive plants its flag on the honest ground this series insists on: there is no setting that gives zero false alarms and zero missed incidents. Tighten the threshold or shorten the persistence and you catch incidents faster while paging on noise; loosen them and the noise stops while real incidents run longer before detection. The exercise has you turn the dial and watch each failure mode arrive, including dropping the threshold until alerts fire on a perfectly healthy history, because reading "there's a tradeoff" convinces no one, and getting paged on noise because you tightened the dial does. The capstone's detection report grades the outcome like an honest scorecard: all four incidents caught (the spike at zero days' lag, drift and cost creep at about two, the quality regression at about four), nothing fired on the clean history, and the lag is printed rather than hidden, because detection lag is the price of not crying wolf and pretending otherwise is how vendors sell dashboards.

#16.6 The flywheel, and the aggregate that lies

Two more ideas complete the practice, one virtuous and one cautionary.

The virtuous one: monitoring is not for admiring dashboards; it is for turning what production teaches you back into fixes and tests. Every refusal, thumbs-down, and terse answer is a free, real-user-flagged example of something your system got wrong. The lab's mining pass surfaces them and clusters them by theme, which is the step that converts noise into a decision: "scattered failures" becomes "904 of these are the mobile app you don't support," which is an actionable sentence with a budget implication. The output is emitted in the Evals dive's JSONL shape, ready for a human to write gold answers and drop into the regression suite, and the loop this closes is the series' largest: evals gate what ships (Chapter 8), production generates failures, observability mines them, the failures become eval cases, the suite gets harder, and quality ratchets. Teams that run this flywheel get better every month for structural reasons, not because anyone is a hero. The honest caveat attached: most failures generate no feedback at all, no thumbs, no ticket, which is exactly why the proxy signals (refusals, drift, sampled judges) exist. You cannot wait to be told.

The cautionary one: the aggregate lies by construction. A global metric can look healthy while one segment burns: one tenant, one region, one plan, one prompt version. The lab's segmentation example stages an enterprise-only latency regression affecting 15% of traffic; the global p95 detector stays silent (the outage dissolves into overall noise) while the enterprise cohort's own p95 triples and screams. The fix is one line of discipline (compute every series per cohort, run the same detectors on each), plus one line of restraint: smaller cohorts are noisier, so slice along the few dimensions that carry genuinely different risk, not every field you log. It is worth knowing that this example itself was rebuilt when the first version's staged incident refused to hide (the honest instance had to be found by measurement, not faked by loosening a threshold); the story is in AUTHORING-LESSONS, and it is the series' methodology applied to its own teaching.

One sidebar earns its place for interview survival: the classic MLOps vocabulary (feature drift, concept drift, SHAP explainability) grew up around tabular predictors with fixed feature vectors and labels that eventually arrive. LLM applications mostly lack those handles, and the mapping is partial: PSI transfers to numeric features, drift-thinking transfers in spirit, and "LLM explainability" as a production practice mostly does not exist yet, whatever a vendor deck implies. Learn the vocabulary; audit the analogies.

#16.7 Where this chapter leaves you

The capstone runs the whole stack as one tool: ingest the history, compute every series, run the tuned detectors, print the dashboard with its incident timeline, and then grade itself against the answer key. That last step is the dive's signature. Monitoring tools normally assert their own usefulness; this one measures it, catch by catch, lag by lag, and stays silent on the healthy history to prove it is not a machine for generating anxiety.

What you take forward is a posture more than a toolkit. Quality is a trend, not a launch checkbox. A number needs a baseline before it means anything. Silent failures need proxy instruments, layered cheap-to-expensive. Alerts spend human trust and must be tuned like it. Aggregates hide cohorts. And production traffic, mined honestly, is the best eval set you will ever own. The industrial versions of every layer exist (OpenTelemetry backends, drift platforms, LLM-native observability products, alert managers), and they will slot into the shapes you built here, which was the reason to build them small first: so the products read as engineering, and their dashboards as claims you know how to check.

Lab manual: README.md · Exercises: EXERCISES.md · Pairs with: Production and Evals