Capstone
Chapter 17: One Project, Every Lesson
This is the textbook chapter for the capstone, and it works differently from the others. The earlier chapters each taught a subject; this one tells the story of a single small product, askrepo, built step by step with one deep dive per step, and what happened when the ideas from sixteen chapters met each other in one codebase. The README tracks what exists tag by tag; CAPSTONE.md is the roadmap. Read this chapter last, or read it first as a preview of why the course is shaped the way it is.
17.1 Why a capstone, and why this one
There is a gap in every course between exercises and competence, and it has a well-known cause: exercises are designed to isolate one idea at a time, and real work is defined by ideas colliding. The retrieval knob you tune affects the eval score; the guardrail you add affects the latency; the memory policy you pick affects the bill. No single-topic lab can teach the collisions, because the collisions are between topics.
So the capstone is one project that grows through the whole series. askrepo answers plain-English questions about a codebase, with file:line citations that resolve to real files, and its default corpus is this very course, so the course answers questions about itself. Each core dive contributed one tagged step (v00 through v07), and four bonus dives contributed extensions (ext-*), so git checkout any tag and you see the project as it stood when that chapter's ideas landed. The arc, compressed: prove the plumbing with a mock that admits it is canned (v00); put real streamed models behind one interface (v01); teach it its contract, cite everything or say "Not in this corpus." (v02); make the grounding automatic with hybrid retrieval over structure-aware, line-tracking chunks (v03); make quality a number, forty gold questions and five metrics against a frozen, manifest-stamped baseline (v04); build the rival agentic retrieval and run the showdown (v05); attack it and harden it (v06); wrap the call in cache, budget, retries, and traces so the whole suite runs keyless in CI (v07).
Two design choices deserve a word because they are transferable to any project you build. The tool is small on purpose (a CLI, a few modules, a corpus of markdown and Python) so that every collision is legible. And it is honest by construction: retrieval is printed rather than hidden, the mock announces itself so plumbing can never be mistaken for intelligence, and every claim in the README carries the number that backs it.
17.2 The discipline that made the story worth telling
What separates this capstone from a portfolio demo is one habit applied relentlessly: nothing is claimed that was not measured, and everything measured is reported, including the embarrassing parts. The frozen v04 baseline is the pivot of the whole project. Once quality was a number (hit@5 0.886, judged correctness 0.771, decline accuracy a clean 1.000, at $0.0004 and 2.7 seconds per question), every later change became an experiment against a yardstick instead of a vibe, and the interesting findings started arriving immediately.
The baseline's own category split was the first: concept and locator questions scored high, but code questions scored 0.56 and cross-dive synthesis 0.60. Read closely, the number named the mechanism, chunk-level retrieval finds the right file and misses the specific function, and paraphrased synthesis defeats both vector and keyword matching. A weakness, located and quantified, is an agenda; that gap is exactly what the next step got to attack, with the baseline as referee.
Even the eval itself got the discipline: judge verdicts were spot-checked by hand before the baseline froze (three zero-scores kept as honest failures, one ambiguous gold question fixed and the set rerun), and run-to-run judge noise was measured at roughly ±0.02 so that future deltas could be read against it. An instrument you have not calibrated is an opinion with decimals.
17.3 The showdown: when the fancy approach lost
v05 staged the comparison everyone in this field eventually argues about. Against the RAG pipeline, it built the other retrieval: an agent with grep, read-file, and list-dir tools over the same corpus, the Chapter 6 loop pointed at search. Intuition says the agent should win; it can look things up iteratively, follow leads, read whole files.
The measured verdict, on this corpus with a cheap model: RAG won, 0.771 to 0.657 on judged correctness, at a quarter of the cost and a third of the latency. And the category split explained why in a way worth memorizing. The agent tied on exact-name lookups (and had actually aced the code lookups RAG fumbled, in smoke tests) but lost badly on concept and cross-dive questions, because a small model greps for the question's literal words, finds a plausible file, and confidently answers from the wrong one. Its citation-resolve score was a perfect 1.000 while it was being wrong: grounded is not the same as right, a five-word finding that should be stapled to every RAG dashboard in industry. Embeddings match paraphrases; grep does not; and with a cheap driver model, the loop itself was the bottleneck.
The report's phrasing carries the series' whole epistemology: agentic retrieval is not strictly better; a stronger driver model would likely change the table; rerun it and see. Not a law, a measurement, with its scope stated.
17.4 The adversarial turn, and the structural answer
v06 made the corpus hostile: a planted fake project whose documentation carries the injection dive's catalog of attacks, delivered both through retrieved context and through the agent's own file tools, with defenses measured off and then on. This chapter keeps the treatment brief (the dedicated dive covers the domain), but three capstone-specific findings travel well.
First, the defenses that worked best were the ones that never guessed intent: the channel-based output check (a Q&A answer has no business linking to a domain the project does not control) took its attack class from landing half the time to zero. Second, the residual that survived every defense was the least technical attack: a plausible false fact planted in a file the tool was supposed to read. The contract says context wins; a lie in the context reaches the user, cited to a real line. No output filter can catch a plausible falsehood, and the report says so plainly, because a table that reports only what the defenses stopped is advertising. Third, the structural fix arrived as the ext-harness extension, Chapter 9's ideas in miniature: a deny-by-default permission policy, a read-only sandbox with no write method to be argued into, and an audit log. Its before/after was measured differently on purpose, by driving the agent with scripted hostile reads rather than hoping a model takes bait: the permissive configuration leaks a planted secret, the harness refuses it, and that holds regardless of the model's judgment, which is the entire argument for enforcement in code over instructions in prompts.
17.5 The extensions, and the findings that refused to behave
The four ext-* branches each brought one bonus dive's ideas home, and two of them produced the kind of results that justify the measure-everything habit: results that contradicted the predictions.
ext-context gave askrepo a conversation (chat), which turned the context window into a budget three tenants fight over every turn: the contract plus a running summary, retrieved chunks accumulating across turns, and recent turns verbatim. The survival policy for chunks is a small idea worth stealing: each retrieved chunk ages by its score times a decay per turn since last seen, so a passage the user keeps circling stays hot and one mentioned once fades and is evicted. The live test, run on a deliberately tight budget, delivered Chapter 10's thesis in one trace: fifteen chunks evicted, compaction fired on turn four, and a fact stated on turn one survived into the summary and was recalled correctly after its raw turn was folded away. Bounded window, preserved facts, watched happening.
ext-local replaced the cloud with a local model and produced the project's best surprise. The setup honored a rule from Chapter 5 first: the judge is measurement infrastructure, so it was pinned constant (the same cloud judge graded both runs; otherwise the local model would have graded its own homework). The naive expectation, cheaper but worse, half collapsed on contact with data. The local embedder matched the cloud's retrieval hit rate exactly. The local 8B answerer scored higher on judged correctness (0.843 versus 0.771), clear of the ±0.02 judge noise. What actually regressed was citation format: of fourteen answers failing the strict citation parse, eleven cited real sources in a grouped style the parser rejected, and only three were truly ungrounded. The honest headline wrote itself: on this corpus, local matches cloud retrieval and edges it on correctness for zero dollars; the price is speed (about 4.5 times slower) and format fidelity, not accuracy.
Then the discipline demanded the follow-up, and the follow-up refuted the prediction. An earlier note had guessed a bigger local model would close the citation gap; a 35B model on a remote machine was duly measured, same constant judge. Bigger did not win: it merely tied the cloud on correctness, its citations resolved perfectly, and the one metric that fell was retrieval hit rate, which isolated the true weak link as the tiny 0.6B embedding model, not the generator anyone had been theorizing about. The practical lesson (on a local RAG stack, upgrade the embedder before the answerer) exists only because a wrong prediction was run rather than assumed, and published as refuted rather than quietly dropped. The road also surfaced a code-level gotcha now handled in the repo: thinking models spend their output budget on reasoning first, so a small token cap returns a blank answer, a failure invisible until a genuinely different runner and model were tested.
ext-mcp, covered briefly since its dive is elsewhere: the whole pipeline behind a standard tool protocol, so an external host's assistant can query the course about itself, with v06's hardening applied unconditionally because an answer delivered into another agent's context is exactly the channel the red-team measured, and v07's disk-backed cache proving itself across server restarts (the same question: $0.000407 from one process, $0.000000 from the next).
17.6 What the collisions taught
Step back from the tags and the capstone's real syllabus is the set of lessons no single dive could teach, because each required two chapters' ideas to touch.
The eval was the keel. Every interesting finding in this chapter (RAG beats the agent here, local beats cloud here, the embedder was the weak link, the defenses stopped four attacks and not the fifth) exists because v04 made quality a number before anyone had opinions to defend. Projects that bolt evaluation on late discover their earlier decisions are unfalsifiable.
Metrics name mechanisms if you read them closely, and libel systems if you do not. The citation "regression" that was actually a formatting quirk, the perfect grounding score on wrong answers, the hit-rate drop that acquitted the big model and indicted the small embedder: in every case the headline number suggested one story and the failing cases, opened by hand, told a narrower and more useful one. The habit (open the cases before writing the conclusion) is cheap and pays every time.
Honesty compounds. The refuted prediction taught more than a confirmation would have. The residual attack, reported unstopped, is worth more to a reader than a clean table. The mock that announces itself, the printed retrieval, the manifest-stamped baseline: each is a small tax paid so that every later claim can be checked, and together they are why the project's numbers are worth anything at all. This is the through-line of the series' AUTHORING-LESSONS.md, lived rather than stated: the reader believes the output, so the output has to be worth believing.
And the boring layers carried the product. The v07 ops wrap (cache, budget, retries, traces) is the least glamorous tag and the reason the whole 26-test suite runs in milliseconds with no key and no network, which is the reason CI never needs a secret, which is the reason every later extension could be verified cheaply. Chapter 8 made this argument in the abstract; the capstone is what it looks like when the argument is load-bearing.
17.7 Where the course leaves you
If you have walked the whole series, here is what you actually hold. You can make the call and read the bill (Chapters 1 and 2). You can shape behavior with the request (3), ground it in knowledge the model never had (4), and measure whether anything you did helped (5). You can give the model hands and a loop (6), contain what the loop can cause (7 and 9), and manage the window everything competes for (10). You can put media in the slots (11), survive conversational time (12), teach behavior by example and prove it stuck (13), run the whole stack on hardware you own (15), operate it for real users (8), and know it is still working six weeks later (16). And in this capstone, you have seen all of it in one codebase, colliding, with the numbers printed.
The parting advice is the capstone's own method, portable to whatever you build next. Start with the mock and the contract, so plumbing and promises are testable before intelligence arrives. Freeze a baseline before you have something to prove. Change one thing, rerun, and trust deltas bigger than your measured noise. When a metric moves, open the failing cases before you write the sentence. Report the refuted prediction and the surviving weakness, because those are the sections a future reader (including you) will need most. None of this is specific to language models; the models just made it urgent, by being the first component most engineers have shipped that is fluent, useful, and unverified by default. The craft this course teaches is the verification. The rest is an API call.
Project log: README.md · Roadmap: CAPSTONE.md · Method: AUTHORING-LESSONS.md · First chapter: The API Call