#Exercises: make the learning stick

dataset → task → scorer → report. The dataset is the hard part: clever scorers and metrics can't rescue an eval built on unrepresentative or mislabeled examples.

Because sarcasm and mixed sentiment ("Not as bad as the reviews said") have no reliable keyword signal; the words point the wrong way. That ceiling is the motivation for an LLM (example 05) and for measuring at all.

exact_match fails (the ! and capital P make the strings unequal); contains_expected passes (case-insensitive substring). Choosing the scorer is choosing what "correct" means, and why example 05's task normalizes the label before scoring.

Recall is 100% (it caught all real spam) but precision is terrible (most flagged emails weren't spam). On an inbox that's mostly ham, plain accuracy could even look bad, and on mostly-spam data, a "flag everything" model could post high accuracy while being useless. Precision/recall/F1 expose what one number hides.

Once the gap shrinks into the margin of error, likely_real flips to False. A small improvement that's inside the noise isn't a result you can trust; you'd need more data or runs to separate it from chance.

Almost certainly not 100%. Errors cluster on the "hard" rows (sarcasm, mixed signals). Some of those labels are genuinely debatable, so a "wrong" answer can be a disagreement with a shaky label, not a model failure. A good eval surfaces that rather than hiding it.

Use a judge when correctness is about meaning and code can't check it (is this summary faithful? is this answer helpful?). The risk: the judge is itself a model with biases and costs a call per grade, so it must be calibrated, not trusted blindly.

B (helpful) under the helpfulness rubric; A (concise) under the conciseness rubric. Same answers, opposite winners, because the rubric defines "better." The rubric is the most important sentence in the eval.

Position bias is a judge favouring whichever answer is shown first. The fix: judge each pair in both orders and only count a win if the same answer wins both ways (otherwise call it a tie). This both detects and neutralizes the bias.

Probably not; the score wobbles run to run. So one number is a point estimate, not the truth. Report a mean with a confidence interval, and to claim system B beats A, run enough times that the difference clears the noise (compare()).

It guessed. It reached the right answer without doing the work (never called the required tool). For an agent, a right answer can hide a broken or unsafe process, so you grade the trajectory (steps + answer) on several axes: correctness, required tool used, no forbidden tools, within the step budget.

Some of that 83% is agreement you'd expect by chance (especially with imbalanced classes). Kappa subtracts the chance baseline, so it's the honest reliability number. A low kappa means your gold labels are noisy, so fix the guidelines and re-annotate before trusting any score built on them.

(1) The difference must clear the margin of error (a small sample can't prove a real gap), and (2) no guardrail metric (latency, refusal rate, cost) may have regressed. A headline win that quietly breaks a guardrail is still a regression.

It can pass correctness (it's true) but fails faithfulness (the context didn't support it; the model made it up and got lucky). That's the danger: an ungrounded answer that happens to be right today teaches you to trust a system that's actually hallucinating. Faithfulness needs no gold answer, only the context, so it catches exactly what correctness misses. The fix at answer time is the grounded prompt: answer only from context, decline when it's silent.

Because run-to-run variation makes small deltas meaningless. compare() only flags a change as real when it exceeds the margin of error, so the tool refuses to cry "regression!" (or "improvement!") over noise. That restraint is what makes an automated gate trustworthy.