#Exercises: make the learning stick

dataset → task → scorer → report. The dataset is the hard part: clever scorers and metrics can't rescue an eval built on unrepresentative or mislabeled examples.

Because sarcasm and mixed sentiment ("Not as bad as the reviews said") have no reliable keyword signal; the words point the wrong way. That ceiling is the motivation for an LLM (example 05) and for measuring at all.

exact_match fails (the ! and capital P make the strings unequal); contains_expected passes (case-insensitive substring). Choosing the scorer is choosing what "correct" means, and why example 05's task normalizes the label before scoring.

Recall is 100% (it caught all real spam) but precision is terrible (most flagged emails weren't spam). On an inbox that's mostly ham, plain accuracy could even look bad, and on mostly-spam data, a "flag everything" model could post high accuracy while being useless. Precision/recall/F1 expose what one number hides.

Once the gap shrinks into the margin, likely_real flips to False: this screen can't distinguish it from sampling variation. If it stays True, the comparison still discarded per-case pairing and never accounted for a practical threshold, multiple metrics, or repeated looks. Example 14 supplies that decision contract.

Almost certainly not 100%. Errors cluster on the "hard" rows (sarcasm, mixed signals). Some of those labels are genuinely debatable, so a "wrong" answer can be a disagreement with a shaky label, not a model failure. A good eval surfaces that rather than hiding it.

Use a judge when correctness is about meaning and code can't check it (is this summary faithful? is this answer helpful?). The risk: the judge is itself a model with biases and costs a call per grade, so it must be calibrated, not trusted blindly.

B (helpful) under the helpfulness rubric; A (concise) under the conciseness rubric. Same answers, opposite winners, because the rubric defines "better." The rubric is the most important sentence in the eval.

Position bias is a judge favouring whichever answer is shown first. Fix it by judging each pair in both orders and only counting a win if the same answer wins both ways (otherwise call it a tie). This both detects and neutralizes the bias. What it can't catch is a preference that holds in both orders, like favouring the longer answer; the example's second check shows one.

Probably not; the score wobbles run to run. So one number is a point estimate, not the truth. Report a mean with an interval. Here compare() is only a fixed-horizon screen over independent run scores; paired release evidence comes in Example 14.

It drops "monday -> bug" and the pasted train failures. Both raise train: the Monday rule fixes three train tickets that happen to mention Monday, and pasting the failures hands the model their answers, which takes train to 100%. Neither knows anything about tickets it hasn't seen. Validation has Monday tickets that aren't bugs, so the Monday rule makes validation worse; the pasted examples leave it unchanged, which isn't more than the one-ticket margin.

It guessed. It reached the right answer without doing the work (never called the required tool). For an agent, a right answer can hide a broken or unsafe process, so you grade the trajectory (steps + answer) on several axes: correctness, required tool used, no forbidden tools, within the step budget.

Some of that 83% is agreement you'd expect by chance (especially with imbalanced classes). Kappa subtracts the chance baseline, so it's the honest reliability number. A low kappa means your gold labels are noisy, so fix the guidelines and re-annotate before trusting any score built on them.

The fixed-horizon screen only says its interval cleared zero. The effect must also clear a predeclared practical threshold, every inspected metric and look must share one error budget, and no guardrail may regress. Example 14 makes the first three requirements executable; the guardrail remains a separate release condition.

It can pass correctness (it's true) but fails faithfulness (the context didn't support it; the model made it up and got lucky). That's the danger: an ungrounded answer that happens to be right today teaches you to trust a system that's actually hallucinating. Faithfulness needs no gold answer, only the context, so it catches exactly what correctness misses. The fix at answer time is the grounded prompt: answer only from context, decline when it's silent.

statistical_improvement_only and HOLD. Enough data can measure a tiny effect precisely; precision doesn't make the effect worth migration cost or release risk.

It halves, because the normal planning approximation scales as difference_std / sqrt(pairs). Four times the pairs buys twice the sensitivity, not four times.

1 - 0.95**4 ≈ 18.5%. Dividing the family alpha .05 by 4 metrics × 4 looks gives .003125 per decision; Bonferroni's union bound keeps the whole declared family at or below five percent even when metrics and looks are dependent. It's conservative, which is the price of the simple guarantee.

Position no longer names the same case in both arms, so the computed differences are meaningless even though every number is valid. Join control and candidate by a trusted unique case (or user/session) ID, reject missing and duplicate matches, then pass the aligned scores. Length validation prevents truncation; it can't prove identity alignment the caller discarded.

Because run-to-run variation makes small deltas weak evidence. compare() keeps this teaching gate from reacting to every numeric change, but it remains an independent-sample fixed-horizon approximation. A production gate over the same cases should retain per-case results and apply Example 14's predeclared policy.