#Exercises: make the learning stick

• RAG / long context → changes what's in the context (knowledge).
• Fine-tuning → changes how the model behaves by default (format, tone, a skill).
• Tools / agents → changes what the model can do (capability).

Most "we need to fine-tune" instincts are actually a knowledge problem (RAG) or a prompt problem (few-shot) in disguise.

RAG, not fine-tuning. The pricing changes and must be cited, and that's knowledge, and baking it into weights would be stale the moment prices change (and can't cite a source). Fine-tuning is for behavior that stays the same, not facts that move.

JSON Lines (JSONL): one conversation per line, each a {"messages": [...]} object. Each example ends on the assistant turn: that's the behavior the model is being taught to reproduce. The system+user turns are the setup; the assistant turn is the lesson.

So you have data the model didn't train on to measure against (Section 7). If you evaluate on the training rows, a model that simply memorized them looks perfect while generalizing terribly; the validation/held-out split is the only thing that catches overfitting.

Any of: malformed/duplicate examples, class imbalance (e.g. 90% of rows are one category, so the model just learns to guess it), a missing assistant turn, a system prompt that drifts between rows, or a token/cost estimate that's bigger than you expected. The script runs all of these on the good set and then on a deliberately broken set so you watch each check fire.

1. upload the training file (files.create)
2. create a job from it (fine_tuning.jobs.create), which is what trains
3. poll the job until done (fine_tuning.jobs.retrieve)
4. use the returned model id

examples/04_run_finetune.py runs this on the mock (instant, free) with nearly identical code to the real OpenAI path.

It does not start a paid job. The real path is opt-in behind --real plus a confirmation, precisely because a fine-tune costs real money and takes real time. You have to mean it. Without --real it runs the safe simulated path.

Almost nothing. It's a normal chat call; you just pass the new model id (e.g. ft:gpt-4o-mini-2024-07-18:...) instead of the base id. There's no special "use a fine-tune" endpoint. The behavior change lives in the weights, not the API shape.

Changed: the tuned model now answers in the rigid category: ... | reply: ... house format the training demonstrated, reliably, with no reminder in the prompt. Didn't change: its underlying knowledge. Fine-tuning taught a format/behavior, not new facts.

Because a model can score perfectly on rows it trained on simply by memorizing them, that number is meaningless. Only data the training never saw (datasets/support_eval.jsonl) tells you whether the behavior generalizes. No held-out score, no claim.

Don't ship it. A fine-tune that doesn't beat the baseline is a regression, no matter how much effort went in. Go back to the dataset (more, cleaner, better- balanced examples) and re-measure. "We already trained it" is the sunk-cost trap.

Overfitting: the model is memorizing the training set instead of learning the general behavior. The first knob to turn down is n_epochs (fewer passes over the data); more/cleaner data helps too. examples/07_hyperparameters.py draws this exact divergence in the simulated loss curve.

The teacher is a big, strong, expensive model that already does the task; the student is a small, cheap, fast model you fine-tune to imitate it. The dataset is cheap because the labels write themselves: you run the teacher over your inputs and use its outputs as the assistant turns. No hand-labeling.

Feed it straight into the Section 5 tune step; it's a normal training file. The example validates it (Section 4) to prove exactly that: a distillation set is just a training set whose labels came from a model instead of a human.

Full fine-tuning updates all the model's weights (heavy: lots of GPU memory). LoRA/PEFT freezes the base weights and trains a small number of added weights and it's far cheaper, almost as good for most tasks. What's identical either way: your dataset. The JSONL you built in Sections 3–4 is exactly what an open-weight trainer consumes too.

Reach for it when there's no single correct output but you can still say which of two is better: tone, conciseness, helpfulness, how firmly to refuse. Its data isn't {prompt → ideal answer}; it's {prompt, chosen, rejected} pairs. DPO nudges the model to make the chosen response more likely than the rejected one. The cheapest source of pairs is your own thumbs up and thumbs down traffic, the feedback loop from the Production dive.

RFT learns from a grader: a function that scores an attempt (unit tests pass, JSON validates, math answer matches, or a rubric judge rates it). There's no target answer and no pair. It beats labeled data when success is easy to verify but hard to demonstrate: many correct outputs exist, so you can check one but can't enumerate them (it's how reasoning models are trained). The risk unique to it: the model hacks a weak grader, passing the check while missing the intent, so the grader needs the same scrutiny as an eval, and you still gate on a held-out set.

The gate decides whether the tuned model provably beat the base model on the held-out set (optionally above a --min-winrate you set). A non-zero exit means it didn't clear the bar, the same signal a CI eval gate gives, so a bad fine-tune can't silently ship.