Core path - 3 of 8
Chapter 3: The Prompt Is the Program
This is the textbook chapter for the Prompt Engineering deep dive. The README is the lab manual; this is the lecture. The first two chapters taught you to send a request. This one is about what the words in that request should say, why word choice changes model behavior at all, and how to tell a genuinely better prompt from one that merely reads better.
3.1 A job title people laughed at
In early 2023, Anthropic posted a job opening for a "Prompt Engineer and Librarian" with a salary range topping out above $300,000, and a good portion of the software industry laughed. Typing questions into a chatbot for an engineer's salary sounded like a parody of the hype cycle. The laughter said more about the laughers than the job. What the posting actually described was someone who could take a fixed, unmodifiable system and reliably get behavior out of it that others could not, then document how, so the result was repeatable.
That is not a new kind of work. It is what a good query optimizer specialist does with a database, or what an old-school compiler engineer did with a temperamental toolchain. The system is fixed; your input is the only lever; expertise is knowing which inputs produce which behaviors and why. The one big idea of this chapter is exactly that:
The model is fixed; the prompt is the program.
You will not touch a weight in this entire dive. Everything you change lives in the request, and the differences you will watch in the lab (a classifier that stops inventing categories, an extraction that stops hallucinating fields, a reasoning problem that goes from wrong to right) come entirely from asking differently.
The job title has since faded, and that is worth understanding too. Prompting did not stop mattering; it stopped being a specialty. It diffused into everyone's job the way "knowing how to search the web" did, and the standalone title dissolved. What remains is the skill, and the skill has more structure than folklore suggests.
3.2 Why does phrasing change anything?
It is genuinely strange that this works. Change a few words in a request and a fixed piece of software behaves differently, sometimes dramatically. No traditional program does that. To use prompting well rather than superstitiously, you need a rough account of why.
Recall what the model is: a machine trained to continue text plausibly, tuned afterward to behave like an assistant. Its training data contained an enormous variety of documents, written by every kind of author, at every level of quality, in every format. When you prompt it, you are not issuing commands to an obedient interpreter. You are establishing a context, and the model produces the kind of text that plausibly follows from that context.
This explains most prompting techniques before you learn them. Why does "You are a senior security engineer reviewing this code" improve a code review? Because text following that framing, in the model's experience of the world, looks like a security engineer's review: it mentions injection risks and unvalidated inputs, because that is what such documents contain. Why do examples work so well? Because few things constrain what comes next as strongly as a visible pattern; three input-output pairs establish a format more firmly than a paragraph of instructions describing it. Why does "answer only from the provided context" reduce invention? Because it reframes the task from "answer from everything you absorbed" to "answer as a careful reader of this document," and careful readers say "the document doesn't mention that."
The research community noticed this property before the public did. The GPT-3 paper in 2020 was titled "Language Models are Few-Shot Learners," and its central finding was that a large enough model could pick up a brand-new task from a handful of examples placed in the prompt, with no retraining. That capability, in-context learning, was not designed in; it emerged from scale, and it surprised the people who built the system. Prompt engineering is the applied craft that grew up around that surprise.
Hold onto one implication, because it separates practitioners from folklore-followers: prompting is empirical. The model's behavior is a fact about a very complicated artifact, not a spec you can reason from. Good prompt engineers behave like experimentalists. Change one thing, observe, keep what works, and (the second half of this chapter) measure rather than squint.
3.3 The core techniques, and what each one actually buys
The lab walks fourteen techniques; this section gives you the map so they cohere rather than accumulate. They cluster into four families, and each family attacks a different failure.
Family one: specify the task. Zero-shot prompting is the baseline: ask clearly, constrain the output, no examples. Most bad outputs trace back to vague requests, and "be specific" sounds too obvious to be a technique until you watch it work. "Summarize this" produces a mush; "summarize this in three sentences for an executive who has not read it, focusing on what changed since last quarter" produces something usable. The model cannot read your mind, and unlike a human colleague it will not ask a clarifying question unless invited; it will fill every unspecified dimension with a guess drawn from the average of the internet.
Few-shot prompting adds worked examples, and it earns its place the moment format and convention matter. Two to five examples teach things that are tedious to state: how to handle edge cases, what counts as "medium" severity, whether dates are ISO or American. The examples do not teach the model new knowledge (it is not learning in any persistent sense); they narrow the space of plausible continuations to the pattern you demonstrated. A practical warning the lab makes vivid: the model imitates your examples' flaws as faithfully as their virtues, and it can overfit to accidents, like always answering "positive" if all your examples happen to be positive.
Family two: give it room to think. Chain-of-thought prompting asks the model to reason step by step before answering, and its discovery has a nice piece of history: researchers at Google reported in 2022 that this one phrase measurably improved performance on math and logic problems, and a follow-up found that even with no examples at all, appending "Let's think step by step" helped. The mechanism is the scratch-space argument from Chapter 2: a model generates one token at a time, and the answer's earlier tokens are the only working memory later tokens get. Force the answer to arrive first and there is nowhere to hold intermediate results; let reasoning come first and the model can lay out the sub-results it needs.
Self-consistency extends this by sampling several independent reasoning paths at higher temperature and taking a majority vote on the final answer. It trades cost (N calls instead of one) for reliability, and it works because errors in reasoning tend to be scattered while correct paths converge. This cost-for-accuracy dial, spend more compute at inference time to get a better answer, is one of the field's recurring moves, and you will see it again in reasoning models.
Family three: control the frame. Role prompting and system prompts set who the model is, who it is talking to, and what the standing rules are. Delimiters and grounding control the boundary between your instructions and the data being processed: fence off the document with clear markers, tell the model to answer only from it, and give it an explicit fallback ("if the context does not contain the answer, say so"). The fallback clause deserves special respect. A model with no permitted way to say "I don't know" will manufacture an answer, because refusing was not on the menu you gave it. Many production hallucination incidents reduce to a prompt that never authorized uncertainty.
You should also hear the foreshadowing: delimiters are the first and weakest defense against a document that contains hostile instructions. The lab plants that flag; Chapter 7 is the war over it.
Family four: compose calls into systems. Prompt chaining splits a job into a pipeline (generate, then critique, then revise), on the reasoning that three focused requests outperform one overloaded one. ReAct interleaves thought, action, and observation so a model can use tools mid-reasoning; done by hand in this lab, it is the visible seed of the agent loop that Chapter 6 grows into the real thing. Reflexion adds a loop of attempt, verify against a real check, reflect, retry, and its lesson generalizes beyond prompting: self-correction works vastly better against an external signal (a test that fails, a parser that rejects) than against the model's own vibes about its answer. Meta-prompting closes the family with a pleasant recursion, using the model itself to rewrite a weak prompt into a strong one. It is genuinely useful as a drafting tool, with the obvious caveat that the output is a draft to test, not a result to trust.
3.4 The anatomy of a reliable prompt
Across the lab's six realistic use cases (support replies, data extraction, code review, summarization, text-to-SQL, classification), the optimized prompts converge on the same skeleton. A reliable prompt tends to answer seven questions:
- Role: who is the model being?
- Task: what exactly should it do?
- Context: what information does it have, clearly fenced off?
- Constraints: what are the hard rules?
- Format: what exact shape should the output take?
- Examples: what does a good answer look like, if format matters?
- Fallback: what should it do when it cannot comply or does not know?
You do not need all seven every time; a quick question needs almost none of them. The checklist is for prompts that will run unattended, thousands of times, against inputs you have not seen. In that setting, every unanswered question is a dimension along which the model will improvise, and improvisation across thousands of runs means inconsistency.
Two habits are worth singling out because they mark the difference between conversational prompting and engineering. First, closed choices beat open ones wherever you can manage it: a classifier given a fixed label set, with an explicit "other" escape hatch, is more reliable than one asked to invent categories, for the same reason multiple-choice exams are easier to grade than essays. Second, even when you constrain the output shape, parse defensively anyway. Structured output support (Chapters 1 and 2) has made "please respond in JSON" mostly obsolete, but belts and suspenders remain the correct fashion in code that runs unattended.
3.5 The advice has a version number
Here is something most prompting guides will not tell you: a chunk of their advice is dated, and the expiry is accelerating.
Chain-of-thought is the clearest case. It was discovered as a prompting trick, became standard advice, and then got absorbed into the models themselves. The reasoning models you met in Chapters 1 and 2 deliberate natively, in a dedicated phase, trained rather than requested. Prompting one of them with "think step by step" scaffolding is at best redundant and at worst counterproductive, like giving turn-by-turn directions to a driver with a navigation system. The current guidance for reasoning models is almost the opposite of the classic guidance: state the goal, the constraints, and the success criteria, then get out of the way.
This does not mean the older techniques were wrong. It means prompting advice is advice about particular artifacts at particular moments, not laws of nature. The durable layer is the understanding underneath: why examples constrain output, why scratch space helps, why unspecified dimensions get filled with guesses, why a model needs permission to say "I don't know." Models will keep changing; those mechanics have held steady across every generation so far. Learn the techniques as instances of the mechanics and you will adapt when the next generation shuffles the specifics, which it will.
There is a second, quieter shift: as models get better, the payoff curve of prompt cleverness flattens. Tricks that doubled accuracy on 2023 models often buy a few points on current ones, because the naive prompt already works. The skill is not becoming worthless; it is moving up a level, from phrasing individual requests to designing the whole context a system presents to a model. The industry has started calling that larger discipline context engineering, and it has its own dive later in this series.
3.6 The capstone is the actual lesson
Every lesson in this lab prints a before and after, and the after usually looks better. Now for the uncomfortable question this whole chapter has been building toward: looks better to whom, on which inputs, and would you bet a production deployment on it?
People are bad judges of prompt quality, in specific and well-documented ways. We evaluate on the three examples we happen to try, which we chose, which resemble each other. We read fluent, confident output as correct output. We remember the impressive win and forget the two quiet regressions. And prompts have a property that makes intuition especially treacherous: a change that improves one kind of input routinely degrades another. Tighten the format rules and the model gets worse at edge cases that do not fit the format. Add an example with a sarcastic review and plain negative reviews start getting flagged as sarcasm. Prompt changes have side effects, and side effects that only show up on inputs you did not test are invisible to eyeballing.
The capstone, optimize.py, is the antidote in its smallest useful form: a labeled set of cases, two prompts, a score for each, and a verdict with a number attached. Running both prompts over thirty cases will never make a conference paper, but it is categorically different from squinting at three outputs, because it can tell you something you did not want to hear. The README makes a prediction worth repeating here: the first time the harness tells you your "improved" prompt actually scored worse, the discipline clicks. Nearly everyone who does this work has that moment, and nobody fully believes in measurement until they do.
That is also why this dive sits where it does in the series. Prompting is the cheapest, fastest lever you have, the bottom rung of the ladder in CHOOSING.md, and you should always try it before reaching for retrieval or fine-tuning or agents. But "try it" only means something if you can tell whether it worked. The capstone is a thirty-line preview of Chapter 5, which turns measurement into a full discipline: judges, metrics, statistical noise, and eval gates in CI. Between here and there, Chapter 4 handles the failure no phrasing can fix, the model that does not know the facts, by changing what you put in the context rather than how you ask.
3.7 Where this chapter leaves you
You should leave this dive with a changed default. Before: "the model gave a bad answer, so the model is bad at this." After: "the model gave a bad answer; what did my request leave unspecified, unfenced, unformatted, or unpermitted?" A surprising fraction of the time, the second question has an actionable answer and the first was simply wrong.
You should also leave with the experimentalist's reflex: one change at a time, observed on more than one input, measured when it matters. That reflex, not any particular trick, is what the expensive job title was really advertising for.
Lab manual: README.md · Exercises: EXERCISES.md · Previous: Claude API · Next: RAG