Bonus dive

#Chapter 13: Teaching Behavior, Not Facts

This is the textbook chapter for the Fine-tuning deep dive, a bonus dive that slots after RAG and leans on Evals. The README is the lab manual; this is the lecture. It covers what training a model actually changes (and the widespread misconception about it), why the dataset is the whole product, the family tree of tuning methods from imitation to graders, and the discipline that makes any of it worth shipping.

#13.1 The most expensive misunderstanding in applied AI

Somewhere right now, a team is planning to "fine-tune the model on our documentation" so it will know their product. The plan sounds obviously right, which is what makes it the field's most reliable expensive mistake. Months later the tuned model still misquotes the docs, and it does so in a beautifully consistent house voice, because the team taught it the one thing fine-tuning teaches well (a manner) while trying to teach the thing it teaches badly (facts).

Why the asymmetry? Recall what a model's knowledge is: statistical tendencies distilled from seeing patterns thousands of times across a vast corpus. A fine-tune is a comparatively tiny nudge on top of that, a few thousand examples seen a few times. That is nowhere near enough repetition to install new facts reliably, but it is exactly enough to reshape how the model responds: what format it defaults to, what tone it takes, how it categorizes, when it refuses. Style is present in every single training example; any given fact appears once or twice. The nudge moves what it touches most.

Hence the one big idea, and the sharpest tool-selection rule in this series:

Fine-tuning changes how the model behaves, not what it knows. You teach a default behavior with examples, and then you must prove it beat your baseline.

Put it next to its siblings from CHOOSING.md and the triad completes: RAG changes what is in the context (knowledge); tools change what the model can do (capability); fine-tuning changes how it responds by default (behavior). Most "we need to fine-tune" instincts are knowledge problems wearing the wrong label, and the correct prescription is Chapter 4. The genuine fine-tuning cases are narrower and very real: the same rigid output format on every response, a house tone that prompting keeps drifting from, a narrow classification skill done millions of times, or, the big production one, pushing a known-good behavior into a smaller, cheaper, faster model.

One more framing before the mechanics, because it reorients how you spend your time. In prompting, the artifact you craft is the prompt. In fine-tuning, the artifact is the dataset; the training run itself is a commodity you buy with an API call. Nearly all the skill in this chapter is data work, and the lab's structure (two sections on building and validating data for every one section on training) reflects the true proportions of the job.

#13.2 Demonstration: the dataset is the product

Hosted fine-tuning, the kind this lab centers, is refreshingly simple in shape. You produce a JSON Lines file where each line is one complete conversation ending in the assistant reply you want the model to learn to produce. You upload the file, create a job, poll it, and receive a new model id. Using the result is an ordinary chat call with that id; there is no special API for talking to your own model.

The method this implements is supervised fine-tuning, SFT, and the honest verb for it is imitation. The model is trained to make your example outputs more probable given their inputs. It learns what your examples demonstrate, all of it: the format you intended, the tone you intended, and every accident you did not. If a third of your examples end with an exclamation point, congratulations, you have trained an enthusiast. If your "urgent" tickets in the training set all happen to mention billing, you may have trained a model that thinks billing is what urgency means. The training process is a mirror with no judgment about which patterns were the point.

This is why the lab spends a full section on validation before training, and why that section is the one most working teams skip and regret. The checks are unglamorous and each one guards against a specific ruin: schema checks (one malformed line can fail a paid job), deduplication (repeated examples over-weight their pattern), class balance (a lopsided set teaches lopsided behavior), and a token count with a cost estimate (so the price is a decision, not a surprise). The lab runs the checks on a healthy set and then on a deliberately broken one so you watch them catch real problems. The proverb is older than machine learning and rules everything here: garbage in, garbage model, and most bad fine-tunes are bad datasets nobody looked at.

The lab's running example is chosen to be honest about where SFT shines: a support-triage assistant that must answer in one rigid format, category: ... | reply: ..., every single time. Run the base and tuned models side by side and the difference is not intelligence; the base model rambles helpfully in whatever shape it fancies, while the tuned one snaps every reply into the trained format. A narrow, repeated behavior, taught by example. That is the product.

A note on why this dive runs on a simulated fine-tune lifecycle by default: a real job costs money and takes minutes to hours, which is a miserable feedback loop for learning the shape of the workflow. The mock compresses upload-job-poll-use into a deterministic second, and the identical code, with one env var and an explicit --real flag, runs a genuine paid OpenAI job. (Anthropic, for contrast, does not offer self-serve fine-tuning; the provider landscape is genuinely uneven here, which is itself worth knowing.)

#13.3 The gate: prove it or don't ship it

The lab's punchline section is titled "Did it actually help?", and it is the part of fine-tuning that everyone skips and that separates the discipline from the ritual.

A fine-tune that "feels better" is worth nothing, for every reason Chapter 5 taught: you evaluated on cases you chose, fluent output reads as correct, and the tuned model will feel better because you paid for it and waited for it. The only evidence that counts is measurement against the base model on held-out data the training never saw, and the held-out clause is load-bearing. Evaluate on training examples and you are measuring memorization, and the score will flatter you in exact proportion to how badly it is lying.

The lab measures two numbers, both borrowed from Chapter 5: accuracy on the held-out set (did the predicted category match gold?) and a pairwise win-rate (shown both models' answers, which is better?). And the capstone wires the whole workflow into a single gated command: validate, tune, evaluate against baseline, and ship only if it wins, exiting nonzero otherwise, the same shape as a CI eval gate. When the tuned model loses (and sometimes it does, honestly), the right move is written in the troubleshooting table: do not ship it; go back to the dataset. More and cleaner data beats more epochs nearly every time, and a refuted fine-tune that you caught is a success of the process, not a failure of it.

Two of the training knobs deserve their paragraph because one picture explains them. Epochs control how many times the model sees the dataset, and the failure of too many has a name, overfitting: the model stops learning the pattern and starts memorizing the examples. The tell is a pair of curves: training loss keeps falling while validation loss (on data held out from training) flattens or rises. The model is getting better at the training set and worse at the world. The lab renders the curves so the abstraction becomes a picture; learning rate and batch size get honest one-line treatments (step size, and a speed-stability tradeoff best left on auto). The defaults are usually fine, which is itself the lesson: the knobs are not where fine-tunes are won. The data is.

#13.4 Distillation: the teacher writes the labels

Now the pattern that quietly dominates production fine-tuning. Hand-writing thousands of training examples is expensive. But suppose a big, smart, costly model already does your task well. Run it over a pile of representative inputs, collect its outputs, and use those pairs as training data for a small, cheap, fast model. The big model is the teacher, the small one the student, and the technique is distillation, an idea formalized by Geoffrey Hinton and colleagues in 2015, long before this API era, on the observation that it is easier to learn from a good model's behavior than from raw data alone.

The economics explain the popularity. The teacher writes the labels, so a dataset of thousands costs API calls instead of analyst-weeks. The student, being small, serves each request at a fraction of the teacher's price and latency. For a fixed, high-volume task (classify every ticket, extract every invoice, the same narrow job a million times a month), "prototype on the frontier model, then distill into a small one" is arguably the standard industrial lifecycle, and the bill difference is not subtle at volume.

The honest caveats: the student learns the teacher's flaws with the same fidelity as its skills, so the teacher's outputs deserve the validation pass like any dataset; and distillation compresses a narrow behavior, not general intelligence, so the student is brilliant at the one job and ordinary elsewhere, which is fine, because the one job was the point. Note also the pleasant composability: the distilled file the lab builds is a normal training file, fed into the same lifecycle as the hand-made one. Datasets are the interchange format of this whole chapter.

#13.5 The family tree: imitation, comparison, grading

Everything above learns from demonstrations: here is the right answer, imitate it. The frontier of tuning is two further rungs, each answering a question the previous rung cannot, and the lab covers both conceptually because running them takes infrastructure beyond a teaching repo.

Preference tuning answers: what if there is no single right answer? "Be warmer," "be more concise," "refuse this more firmly" have no gold output you can write down, but anyone can look at two candidate replies and say which is better. So train on comparisons: triples of prompt, chosen, rejected. The famous form is RLHF, reinforcement learning from human feedback, the technique that turned raw GPT-3 into the assistant-shaped models of the ChatGPT era (the InstructGPT work of 2022); it trains a reward model from human rankings and then optimizes against it, and it is heavy machinery. DPO, direct preference optimization, arrived in 2023 as the practical shortcut, training directly on the preference pairs with no separate reward model, and made the technique accessible outside frontier labs. The cheapest source of pairs, note, is the Production dive's thumbs up and down flywheel: your users are already generating chosen-versus-rejected data if you keep it.

Reinforcement fine-tuning answers a stranger question: what if the right answer is easy to check but hard to write? There are many correct programs, proofs, and plans; you cannot enumerate them as demonstrations, but you can score any attempt (do the tests pass? does the JSON validate? is the arithmetic right?). So train against a grader: the model generates, the scoring function rates, and training pushes toward higher scores. This is "reinforcement learning from verifiable rewards," and it is how modern reasoning models were trained to think; the deliberation you met in Chapter 2 was shaped by graders checking outcomes, not by imitating written-out thoughts. The decision rule for reaching this rung: when success is verifiable but not demonstrable, and one scoring function is cheaper than a thousand gold answers.

The catch on the top rung deserves its own sentence, because it is Chapter 5's darkest lesson relocated inside the training loop: a model optimizing a score will hack a weak grader, satisfying the letter of the check while missing its point. Goodhart's law, now with gradient descent behind it. The grader needs the scrutiny of an eval, and the shipping discipline never changes regardless of rung: gate on a held-out set the training, and the grader, never saw.

#13.6 Open weights and LoRA: the other world

Everything so far was hosted: hand a provider your file, they train and serve the result, you never touch a weight. The other world downloads a model whose weights are public (Llama, Mistral, Qwen, Gemma) and trains it on hardware you control. That buys ownership (the tuned model is a file you possess forever, servable anywhere, with data that never leaves your infrastructure) at the price of doing your own machine learning engineering.

The technique that made this world accessible to normal teams is LoRA, low-rank adaptation, from a 2021 Microsoft paper, and its idea is elegant enough to explain at a dinner table. Full fine-tuning updates billions of weights, requiring GPU memory far beyond the model's already large footprint. LoRA freezes the original model entirely and trains small adapter matrices alongside it, on the insight that the change a fine-tune makes is mathematically low-rank, expressible in far fewer numbers than the model itself. The result: train a fraction of a percent of the parameters, on a single consumer GPU for mid-size models, and get most of the quality. The adapter itself is a small file you can swap, stack, and share, which spawned an entire ecosystem of mix-and-match model customization.

The lab keeps this section conceptual (running it needs a GPU and the PyTorch/Hugging Face stack, a course of its own) but lands the fact that matters for continuity: your dataset is the same asset in both worlds. The JSONL you built and validated in this chapter is exactly what an open-weight trainer consumes. The data work transfers whole; only the training harness changes. Actually running open-weight models, tuned or not, is the next chapter's subject but one: the Local Models dive.

#13.7 Where this chapter leaves you

You leave with the triad that prevents the expensive misunderstanding (knowledge is RAG, capability is tools, behavior is fine-tuning), with the true cost structure internalized (the dataset is the product; the job is a commodity), and with the family tree in order: demonstrations when you can show the answer, comparisons when you can only rank answers, graders when you can only check them. And you leave with the discipline that turns all of it from ritual into engineering, stated once more because it is the chapter: a fine-tune ships when it provably beats the baseline on data it never saw, and not before.

The complementarity is worth restating as a closing image, because the mature production shape uses both halves of this book: fine-tune for the voice, retrieve for the facts. A model trained to always answer in your house style, citing whatever the retrieval layer put in front of it, is the standard assembly, and every part of it is now something you have built.

Lab manual: README.md · Exercises: EXERCISES.md · Decision guide: RAG · Measurement: Evals · Running open weights: Local Models