#Exercises: make the learning stick

• RAG / long context → changes what's in the context (knowledge).
• Fine-tuning → changes how the model behaves by default (format, tone, a skill).
• Tools / agents → changes what the model can do (capability).

Most "we need to fine-tune" instincts are actually a knowledge problem (RAG) or a prompt problem (few-shot) in disguise.

RAG, not fine-tuning. The pricing changes and must be cited, and that's knowledge, and baking it into weights would be stale the moment prices change (and can't cite a source). Fine-tuning is for behavior that stays the same, not facts that move.

JSON Lines (JSONL): one conversation per line, each a {"messages": [...]} object. Each example ends on the assistant turn: that's the behavior the model is being taught to reproduce. The system+user turns are the setup; the assistant turn is the lesson.

So you have data the model didn't train on to measure against (Section 7). If you evaluate on the training rows, a model that simply memorized them looks perfect while generalizing terribly; the validation/held-out split is the only thing that catches overfitting.

Any of: malformed/duplicate examples, class imbalance (e.g. 90% of rows are one category, so the model just learns to guess it), a missing assistant turn, a system prompt that drifts between rows, or a token/cost estimate that's bigger than you expected. The script runs all of these on the good set and then on a deliberately broken set so you watch each check fire.

1. upload the training file (files.create)
2. create a job from it (fine_tuning.jobs.create), which is what trains
3. poll the job until done (fine_tuning.jobs.retrieve)
4. use the returned model id

examples/04_run_finetune.py runs this on the mock (instant, free) with nearly identical code to the real OpenAI path.

It does not start a paid job. The real path is opt-in behind --real plus a confirmation, precisely because a fine-tune costs real money and takes real time. You have to mean it. Without --real it runs the safe simulated path.

Almost nothing. It's a normal chat call; you just pass the new model id (e.g. ft:gpt-4o-mini:...) instead of the base id. There's no special "use a fine-tune" endpoint. The behavior change lives in the weights, not the API shape.

Changed: the tuned model now answers in the rigid category: ... | reply: ... house format the training demonstrated, reliably, with no reminder in the prompt. Did not change: its underlying knowledge. Fine-tuning taught a format/behavior, not new facts.

Because a model can score perfectly on rows it trained on simply by memorizing them, that number is meaningless. Only data the training never saw (datasets/support_eval.jsonl) tells you whether the behavior generalizes. No held-out score, no claim.

Don't ship it. A fine-tune that doesn't beat the baseline is a regression, no matter how much effort went in. Go back to the dataset (more, cleaner, better- balanced examples) and re-measure. "We already trained it" is the sunk-cost trap.

Overfitting: the model is memorizing the training set instead of learning the general behavior. The first knob to turn down is n_epochs (fewer passes over the data); more/cleaner data helps too. examples/07_hyperparameters.py draws this exact divergence in the simulated loss curve.

The teacher is a big, strong, expensive model that already does the task; the student is a small, cheap, fast model you fine-tune to imitate it. The dataset is cheap because the labels write themselves: you run the teacher over your inputs and use its outputs as the assistant turns. No hand-labeling.

Feed it straight into the Section 5 tune step; it's a normal training file. The example validates it (Section 4) to prove exactly that: a distillation set is just a training set whose labels came from a model instead of a human.

Full fine-tuning updates all the model's weights (heavy: lots of GPU memory). LoRA/PEFT freezes the base weights and trains a small number of added weights and it's far cheaper, almost as good for most tasks. What's identical either way: your dataset. The JSONL you built in Sections 3–4 is exactly what an open-weight trainer consumes too.

Reach for it when there's no single correct output but you can still say which of two is better: tone, conciseness, helpfulness, how firmly to refuse. Its data isn't {prompt → ideal answer}; it's {prompt, chosen, rejected} pairs. DPO nudges the model to make the chosen response more likely than the rejected one. The cheapest source of pairs is your own 👍/👎 traffic (the Production feedback flywheel).

RFT learns from a grader: a function that scores an attempt (unit tests pass, JSON validates, math answer matches, or a rubric judge rates it). There's no target answer and no pair. It beats labeled data when success is easy to verify but hard to demonstrate: many correct outputs exist, so you can check one but can't enumerate them (it's how reasoning models are trained). The risk unique to it: the model hacks a weak grader, passing the check while missing the intent, so the grader needs the same scrutiny as an eval, and you still gate on a held-out set.

The gate decides whether the tuned model provably beat the base model on the held-out set (optionally above a --min-winrate you set). A non-zero exit means it did not clear the bar, the same signal a CI eval gate gives, so a bad fine-tune can't silently ship.