Bonus dive

#Chapter 16: The Next Six Weeks

This is the textbook chapter for the Observability deep dive, a bonus dive that pairs with Production and Evals. The README is the lab manual; this is the lecture. It covers the failures that never throw an exception, the handles you actually have when your system's core component takes free text and returns free text, the craft of alerting that catches incidents without training everyone to ignore it, and where the industry standard (OpenTelemetry) takes over from what you built by hand.

#16.1 Nothing turned red

Here's a failure story that has happened, in some variation, to nearly every team running a language model application, and will happen to yours.

The launch went well. The eval suite passed, the demo impressed, the dashboards came up green. Weeks pass. Then a slow realization assembles itself from fragments: support tickets sound frustrated, a customer mentions offhand that "the assistant got worse," someone finally reads a sample of recent transcripts and winces. The investigation finds the cause was weeks old. Maybe users had gradually started asking about a product area the knowledge base never covered, and the model had been gamely answering badly the whole time. Maybe the provider had swapped the model version underneath, and answers got terser. Nobody was paged, because nothing failed. No exception, no error rate, no red. The system did exactly what it was built to do, fluently, while its quality rotted.

Traditional software mostly fails loudly: exceptions, timeouts, 500s, all of them countable, all of them wired to pagers decades ago. Language model applications add a class of failure that is silent by construction, because the system's whole job is producing plausible text, and it'll keep doing so at exactly the same latency, cost, and error rate while being wrong more often. Your existing dashboards can't see it. That isn't a gap in your dashboards; it's a property of the workload, and it needs its own instruments.

Hence the one big idea, positioned deliberately against the rest of this series:

A prototype is judged once. A production system is judged continuously, so your quality has to be a trend you watch, not a number you checked at launch.

The Evals dive (Chapter 5) told you a change was better today, on the questions you had today. Production (Chapter 8) told you what one request did. This dive is about the six weeks after, and it completes a triangle those two chapters left open.

#16.2 Learning detection honestly: the simulator

This dive needed history to teach on (you can't learn to spot a trend from one request), so its equivalent of Production's mock model is a deterministic traffic simulator: forty-two days of realistic request logs for the same support assistant, with four real incidents buried inside. Users drift toward asking about an unsupported mobile app. A silent model swap makes answers terser for a stretch, then rolls back. A prompt change bloats the context and cost creeps. And one day, latency spikes and recovers, a deliberate red herring to teach the difference between a blip and a regression.

Two design decisions in that simulator carry more pedagogy than anything else in the repo, and both are principles this series' AUTHORING-LESSONS.md formalized. First: the log records contain only what a real system would observe (question, latency, cost, tokens, whether it refused) and no "was this answer good?" field. The ground-truth incident schedule exists separately, as a private answer key used only to grade your detectors at the end. Leak the label into the data and every detector becomes a fake that reads the answer instead of inferring it. Second: the incidents are injected by genuinely changing the simulated behavior, never by stamping numbers into a chart. The quality regression makes the answers actually terser and more evasive, so a judge scores them lower because they're worse; the cost creep adds real prompt tokens. A detector that works here works on reality, because the signal falls out of behavior, not out of a rigged flag.

Sit with the absence of that quality field for a moment, because it's the entire problem statement. In classic machine learning operations, labels eventually arrive (the loan defaulted or didn't) and you can score yesterday's predictions. In an LLM application, almost no request ever gets a verdict. Everything that follows is a strategy for inferring health from what you do have.

#16.3 Numbers need context: metrics and baselines

The first handles are the operational ones, cheap because they're already in your logs (the exact trace shape Production emits, which is the point of the pairing): request volume, latency, cost per request, error and refusal rates, cache hit rate.

Two habits turn those from data into instruments. The first is an old operations lesson: watch percentiles, not averages. The p50 tells you the typical experience; the p95 tells you what your unluckiest users feel, and the tail is where trouble shows first while the average smiles blandly over it. The second is this dive's methodological spine: a number means nothing alone. "Cost per request is $0.000050" is noise; "up from a baseline of $0.000023, a shift many standard deviations beyond normal variation" is an incident. The lab's machinery learns what normal looked like from a clean baseline window, then scores every new day as a z-score, how many baseline standard deviations from usual. The elegance is that z-scores are unitless: the same "about three sigma is weird" intuition works for latency, cost, and refusal rate alike, with no hand-tuned threshold per metric. You'll meet this trick everywhere real monitoring is done, because it converts every metric into the same question: how surprising is today?

#16.4 The two drifts

The silent failures come in two species, and they need different detectors; a single "quality number" would conflate exactly the things you need to tell apart.

Input drift is the world changing: users start asking things your system was never built for. Nothing errors, because answering badly isn't an error. The lab detects it three ways, deliberately cheapest first. Novel-term rate is pure string counting: what fraction of today's questions use words the baseline never saw? (When a company launches a mobile app and the assistant's corpus knows nothing about it, "iPhone" and "notification" light this up within days.) Embedding drift asks the same question in meaning-space, using Chapter 4's machinery for a new purpose: how far has today's center of mass moved from the baseline's? And PSI, the population stability index, is the classic distribution-shift statistic from credit-risk modeling, shown here on a numeric feature so you've seen the real thing in its native habitat. The layering is the lesson: reach for string counting before embeddings, and embeddings before statistics, because the cheap detector you actually run beats the sophisticated one you meant to set up.

Quality drift is your system changing: same questions, worse answers, classically because the provider updated the model underneath you (a thing that genuinely happens, and which pinned model versions only partly protect against). Latency, cost, and errors never move. The only instrument that sees it is judgment, and judgment costs money, so you sample: a slice of each day's answers, scored by a judge (Chapter 5's LLM-as-judge, or the lab's deterministic rule-based stand-in). And because a mean over twenty sampled answers is a point estimate, the lab reports it with a confidence interval and enforces the discipline that follows: a dip inside the error bars isn't a regression yet. Chapter 5's statistics weren't academic; here they're the difference between paging someone at 2 a.m. and waiting for evidence.

The lab has you watch both detectors run against the same history: the sampled quality score falls clear of its error bars during the model swap and recovers at rollback, while refusal rate does its own unrelated dance during the input drift. Two diseases, two instruments, separable diagnoses. That's what "the metric that moved names the stage that broke" (a lesson from this series' own capstone postmortems) looks like as a monitoring practice.

#16.5 Alerting: the craft of being told

A dashboard nobody watches is a decoration; the point of all this is to be told. And alerting is where monitoring becomes a human-factors discipline, because the failure mode isn't technical, it's psychological, and it has a documented body count in other fields: hospital alarm systems fire so often that clinicians tune them out, and the phenomenon (alarm fatigue) has contributed to real patient deaths. Operations teams are the same animal. The naive alert (page if p95 > 300ms) fires on ordinary bad Tuesdays until someone mutes it, and the muted alert is the one that sleeps through the real outage. Every alert that wastes an engineer's trust withdraws from an account that funds the response to the one that matters.

The lab's detector is built from three parts that together encode the craft. A baseline z-score (how surprising is today, in units of normal variation). A direction (up is bad for latency; down is bad for a quality score; a metric improving should never page). And persistence: require the breach to hold for N consecutive days before paging. Persistence is the dial that separates the simulator's one-day latency spike (which shouldn't page as a regression, and doesn't) from the multi-week drifts (which should, and do).

And here the dive plants its flag on the honest ground this series insists on: there's no setting that gives zero false alarms and zero missed incidents. Tighten the threshold or shorten the persistence and you catch incidents faster while paging on noise; loosen them and the noise stops while real incidents run longer before detection. The exercise has you turn the dial and watch each failure mode arrive, including dropping the threshold until alerts fire on a perfectly healthy history, because reading "there's a tradeoff" convinces no one, and getting paged on noise because you tightened the dial does. The capstone's detection report grades the outcome like an honest scorecard: all four incidents caught (the spike at zero days' lag, and drift, cost creep, and the quality regression at two each), nothing fired on the clean history, and the lag is printed rather than hidden, because detection lag is the price of not crying wolf and pretending otherwise is how vendors sell dashboards.

#16.6 The feedback loop, and the aggregate that lies

Two more ideas complete the practice, one virtuous and one cautionary.

The virtuous one: monitoring isn't for admiring dashboards; it's for turning what production teaches you back into fixes and tests. Every refusal, thumbs-down, and terse answer is a free, real-user-flagged example of something your system got wrong. The lab's mining pass surfaces them and clusters them by theme, which is the step that converts noise into a decision: "scattered failures" becomes "900 of these are the mobile app you don't support," which is an actionable sentence with a budget implication. The output is emitted in the Evals dive's JSONL shape, ready for a human to write gold answers and drop into the regression suite, and the loop this closes is the series' largest: evals gate what ships (Chapter 5), production generates failures (Chapter 8), observability mines them, the failures become eval cases, the suite gets harder, and quality only moves one way. Teams that run this loop get better every month for structural reasons, not because anyone is a hero. The honest caveat attached: most failures generate no feedback at all, no thumbs, no ticket, which is exactly why the proxy signals (refusals, drift, sampled judges) exist. You can't wait to be told.

The cautionary one: the aggregate lies by construction. A global metric can look healthy while one segment burns: one tenant, one region, one plan, one prompt version. The lab's segmentation example stages an enterprise-only latency regression affecting 15% of traffic; the global p95 detector stays silent (the outage dissolves into overall noise) while the enterprise cohort's own p95 nearly quadruples and screams. The fix is one line of discipline (compute every series per cohort, run the same detectors on each), plus one line of restraint: smaller cohorts are noisier, so slice along the few dimensions that carry genuinely different risk, not every field you log. It's worth knowing that this example itself was rebuilt when the first version's staged incident refused to hide (the honest instance had to be found by measurement, not faked by loosening a threshold); the story is in AUTHORING-LESSONS, and it's the series' methodology applied to its own teaching.

One sidebar is here for interview survival: the classic MLOps vocabulary (feature drift, concept drift, SHAP explainability) grew up around tabular predictors with fixed feature vectors and labels that eventually arrive. LLM applications mostly lack those handles, and the mapping is partial: PSI transfers to numeric features, drift-thinking transfers in spirit, and "LLM explainability" as a production practice mostly doesn't exist yet, whatever a vendor deck implies. Learn the vocabulary; audit the analogies.

#16.7 The standard underneath: OpenTelemetry and OTLP

Everything to this point analyzed logs the application had already written, which is the right order to learn in: you can't judge a telemetry product until you know what you'd do with the data yourself. But no team ships the from-scratch version, and the reason isn't that the code is hard. It's that telemetry has a standard, and standards buy things that hand-rolled formats can't.

OpenTelemetry is that standard, and OTLP is its wire protocol. The lab's final section emits the same LogRecord you've been reading all along as real OTel spans and metrics, posts them as protobuf over OTLP, and receives them at the other end with a receiver small enough to read in one sitting, so the protocol stops being a logo on an architecture diagram and becomes bytes you watched cross a socket. Three things come with adopting it. A wire format every backend accepts, so switching from Jaeger to Honeycomb to Datadog is a URL and an auth header rather than a re-instrumentation project. Context propagation, meaning a trace id that survives across services, threads, and queues, so one user request stays one story even when six processes touched it. And instrumentation you didn't write: the HTTP client, the database driver, and the model SDK all have community instrumentation that emits into the same trace for free.

Two details in that section carry more weight than their size. The first is semantic conventions: attribute names like gen_ai.request.model and gen_ai.usage.input_tokens are agreed across languages and vendors, which is what lets a backend render a model call view without knowing anything about your app. They're also still experimental, every single one of them, having been renamed more than once and having moved to a repository of their own in mid-2026, and there's deliberately no conventional attribute for cost, because cost is priced per vendor, per model, per contract, and the standard refuses to guess. So cost goes in your own namespace, which is the general rule worth taking: convention names for conventional things, your own prefix for everything else, and never a private name for something the convention already covers.

The second is the difference between a span and a metric, which the lab makes numeric: 300 replayed requests produce 300 spans and 8 metric points, and at 300 million requests it's still 8 metric points, because metrics are aggregated inside the process before they're shipped. That asymmetry is the whole economics of production telemetry. Spans are rich, sliceable, and priced per event, so at volume you sample them; metrics are cheap and complete, so you keep them at 100%. Hence the operating rule, alert on metrics, debug on traces, and its corollary about cardinality: a trace id on a span costs nothing, while the same id as a metric attribute creates one time series per request and turns a cheap instrument into an unbounded bill.

The failure modes are worth memorizing because they're silent, and because the folklore about them is unreliable enough that the lab makes you probe each one rather than take it on faith. Spans are batched rather than sent as they end, so when the queue drains is a real question; Python's SDK registers an atexit hook and flushes on a clean exit, which is kinder than most write-ups (and an earlier draft of this chapter) claim, but no hook survives an exit that skips atexit, meaning os._exit, a SIGKILL, an OOM kill, or a container stopped past its grace period. Compression is off by default in that same SDK, which is worth roughly a sevenfold difference on a bill metered by ingest volume. Backends have look-back windows, so backfilled history that is weeks old can be accepted and never displayed. And message content, the question and the answer, is a PII decision rather than a debugging convenience: it ships user text into a third-party store on someone else's retention schedule, which is why every serious GenAI instrumentation gates it behind a flag that defaults to off, and why most teams enable it for a sampled slice only. And the quietest failure of all is instrumentation rot: rename an attribute and nothing throws, the spans keep flowing, and every dashboard and alert keyed to the old name goes blank without a single error to investigate. Instrumentation is therefore code that needs tests like any other, pinning the attribute names and the metric shape, which is the one test in the lab worth copying wholesale into your own work.

What the section is really for, though, is the boundary it draws. Adopting OpenTelemetry replaces the parts of this chapter that were plumbing: writing telemetry down, and computing metrics from it yourself. It replaces none of the judgement. A backend will happily store a million perfectly formed spans and never once tell you that quality drifted, because baselines, drift detection, sampled judges, alert tuning, and the mining loop are decisions about meaning, and no wire protocol carries meaning. OTel is transport, not judgement. That sentence generalizes past this chapter, to every "just use the industry tool" upgrade in the series: the tool buys you the plumbing, reliably and cheaply and better than you'd have built it, and leaves the interpretation exactly where it was, with you.

#16.8 Where this chapter leaves you

The capstone runs the whole stack as one tool: ingest the history, compute every series, run the tuned detectors, print the dashboard with its incident timeline, and then grade itself against the answer key. That last step is the dive's signature. Monitoring tools normally assert their own usefulness; this one measures it, catch by catch, lag by lag, and stays silent on the healthy history to prove it isn't a machine for generating anxiety.

What you take forward is a posture more than a toolkit. Quality is a trend, not a launch checkbox. A number needs a baseline before it means anything. Silent failures need proxy instruments, layered cheap-to-expensive. Alerts spend human trust and must be tuned like it. Aggregates hide cohorts. And production traffic, mined honestly, is the best eval set you'll ever own. The industrial versions of every layer exist (drift platforms, LLM-native observability products, alert managers), and they will slot into the shapes you built here, which was the reason to build them small first, so the products read as engineering, and their dashboards as claims you know how to check. One of those layers you didn't merely map but actually crossed: the telemetry itself now leaves the process as real OpenTelemetry over OTLP, which means the from-scratch stack and the industry standard have already been made to meet, and you know precisely which half of the problem each of them solves.

Lab manual: README.md · Exercises: EXERCISES.md · Pairs with: Production and Evals