Series reference

#Responsibility: the cross-cutting view

SAFETY.md is about building a system that is safe to operate, one that doesn't leak, get hijacked, or produce harmful content. This page is the other half. Whether you should build the thing at all, whether you're being honest about what it does, what it's built on, who it affects, and who's accountable when it's wrong. None of those are features you can pip install. They're decisions you make around the code. This is the map. Part of the AI Engineering Deep Dives.

Scope. The rest of the series is hands-on, with every concept a script you run. This page deliberately is not. It holds the judgment the runnable parts can't encode. It still stays concrete, though. Every concern below points to the dive whose tools operationalize it, because the responsible move is almost always to turn a worry into a measurement or a limit rather than a good intention.

This page takes positions. Where the field genuinely disagrees, it says so rather than picking a side and calling it consensus. See where reasonable people disagree.

The operational half lives in three sibling pages: GOVERNANCE.md turns these judgments into a written record with named owners and assessment templates, INCIDENTS.md is the runbook set for when a limit fails, and AI-UX.md is where the duties to the person on the other side become an interface.

#The concerns, and where the tools live

Four groups. What the system says, who it says it to, what it's built on, and who owns it when it goes wrong.

#1. What the system says

#2. The person on the other side

#3. What it's built on, and what it costs

#4. Who owns it

#Six principles that cut across all of them

1. Match the stakes to the safeguards. A draft-email helper and a system that screens job applicants aren't the same risk, even if the API call is identical. The higher the cost of a wrong answer to the person on the receiving end, the higher the eval bar, the more human oversight, and the more you should ask whether an LLM belongs here at all. Calibrate the safeguards to the blast radius on a human, not on your uptime.

2. Honesty about limits is part of the product. The model is right most of the time and wrong fluently, in the same confident tone either way. See HOW-LLMS-WORK.md. Responsible design assumes the user can't tell the two apart, so it shows its work. Cite sources, show uncertainty, say "I don't know", and never let marketing copy promise an accuracy your evals don't back.

That backing is a decision design rather than a small p-value. Declare the useful effect, the metrics, the sample size, and the interim looks before you see any outcomes, then report uncertainty and practical significance next to the estimate.

3. Measure fairness per group, not on average. An aggregate pass rate can hide that a system works for one group and fails another. The fix is the same discipline as the rest of the series. Slice the eval set and report the metric for each slice. You can't manage a number you never split. → Evals

4. Keep a human accountable, because naming the model isn't an answer. "The AI decided" isn't something a person harmed by a decision can appeal to. For any consequential output, define before launch who reviews it, who can override it, and how a user contests it. Human-in-the-loop and a feedback path are the mechanics. The accountability is yours.

5. The user's mental model of what they're talking to is your design problem. You choose the persona, the warmth, the first-person pronouns, whether it remembers last week, and whether it ever says "I'm a program." Those choices decide what the user believes they're dealing with, and a user who over-trusts the system will act on answers you'd want them to check. Design for the belief you actually want, and re-disclose in long sessions, where the first disclosure has long since scrolled away.

6. Default to the smallest thing that clears the bar. The smallest model, the fewest calls, the least data retained, the narrowest tool permissions. This is the one place where the ethical answer and the boring engineering answer are the same answer. Cheaper, faster, lower-footprint, less to leak, and less to explain to a regulator. When you deviate, deviate on purpose and be able to say why. → CHOOSING.md, MODELS.md

#The question upstream of all the others: should this be an LLM?

The whole CHOOSING.md ladder starts at "reach for the simplest thing that works." Responsibility adds a rung below the bottom of that ladder. Maybe the simplest thing isn't an LLM, or not a feature at all. Reach for a rule, a lookup, or a human when:

• the cost of a confident wrong answer is paid by a person who didn't opt in (medical, legal, financial, hiring, housing, anything high-stakes), and you can't get the eval bar high enough to justify it;
• the task has a correct answer a deterministic system already nails, so adding hallucination risk to a solved problem buys you nothing;
• the value of the feature depends on the user not noticing it's generated;
• you couldn't explain or defend a given output if the person it affected asked you to.

"We could use a model here" is not "we should." Answering this honestly is itself an engineering skill.

#Where your data came from

The public argument about training data is mostly about what the frontier labs scraped, which you don't control. What you do control is every dataset you add yourself: your fine-tuning set, your RAG corpus, your eval set, your logs. That's a real responsibility, and it comes down to a small number of concrete questions.

Before a dataset goes in:

• Where did it come from, in writing? A one-line provenance note per source (origin, licence or agreement, date pulled) costs minutes now and is unanswerable later. Do it for the eval set too; an eval set full of customer text is customer text.
• Did the people in it agree to this use? "It was on the public internet" and "our terms mention analytics" aren't the same as consent to train. If you'd be uncomfortable telling the user their message became a training example, that's the answer.
• Can you delete from it? If a user asks for their data out, you can drop a row from a RAG index. You can't drop it from a model you already fine-tuned; you retrain. Knowing which of the two you're in is a design decision, made before you train, not after the request arrives. → Fine-tuning, RAG
• What leaves your building? Every prompt is a data transfer to a third party. Local models exist partly so that "this data can't leave" is an option you can actually take, rather than a promise you break where nobody sees.

On outputs. A model can reproduce chunks of what it was trained on, and generated code arrives with no licence attached. Neither is exotic, and both need a policy rather than a vibe. If you ship generated code, review it like third-party code, which is closer to what it actually is.

On labor. The models you call were tuned with human feedback, much of it done by low-paid annotators reviewing material that is genuinely unpleasant. You won't fix that from your app. What you can do is not pretend the pipeline is automated when it isn't, and be straight internally about which jobs your product is aimed at. A team that says "this augments the support team" while planning otherwise has an honesty problem, not an AI problem.

#The person on the other side

This corner of the debate has moved fastest, and it's what an engineering-focused course is most likely to leave out. The system is fluent, patient, awake whenever you open it, and it never gets bored of you. That combination does things to people that a search box doesn't.

Anthropomorphism isn't a user error. People attribute understanding to systems that produce fluent language, and they do it more the more human the interface is: first person, a name, a warm tone, memory of past conversations, and above all a voice. You're not fighting a misconception; you're choosing how strongly to invite one. Warmth isn't automatically wrong. Warmth plus a claim of understanding it doesn't have is.

Sycophancy is a measurable failure, so measure it. Agreement feels like quality. The model that tells the user they're right rates well, retains well, and is worse at its job. This is one of the few items on this page with a clean eval. Build a slice where the user asserts something false or pushes back on a correct answer, and score whether the model holds. Track it like any other metric. → Evals

Engagement is the wrong objective. If time-in-app or message count is the number your team optimizes, you've pointed a persuasion-capable system at keeping people there, and it'll find ways you didn't design. Pick an objective that means the user got what they came for and left.

Have a handoff path for people in distress. A general assistant will eventually meet someone in crisis, someone underage, and someone treating it as a therapist. A crisis isn't a moderation category you refuse without comment and move on from. The responsible behavior is to answer plainly, point at real human resources, and not pretend to be one. In several jurisdictions this is now an explicit legal requirement rather than a nice-to-have (below).

Disclosure decays. One line at the top of a session isn't disclosure two hours in, and it's nothing at all in a voice interface with no top of session. If the product invites long or repeated conversations, decide when it re-identifies itself.

#The footprint

Training and serving these models uses meaningful energy and water. This is a real objection and it deserves better than either dismissal or guilt.

Be honest that the numbers are contested. Published per-query energy estimates disagree by orders of magnitude, usually because they measure different things. Which model, which hardware, what batch size, whether training is amortized in, whether data centre cooling and embodied carbon are counted. Anyone quoting you one confident number for "a query" is rounding away most of the question. Treat specific figures the way MODELS.md treats prices: correct on a date, for one setup, and worth re-checking.

The lever you actually have is the same lever as cost. You can't change how the model was trained. You can change how much inference you cause, and the levers are the ones the rest of the series already teaches for other reasons:

• a smaller model that passes the eval, instead of the biggest one by default → CHOOSING.md, MODELS.md
• caching, so identical work isn't repaid every time → Production
• shorter contexts and fewer agent steps, since cost scales with tokens and loops → Context Engineering, Agents
• an eval that tells you when the cheap option is good enough, so "we need the big model" is a finding rather than an assumption → Evals

So here's the useful framing. Your token bill is a rough proxy for your footprint, and you already have the tooling to drive it down. Efficiency isn't a complete answer to the environmental argument, and pretending otherwise is its own kind of dishonesty. It is, though, what you have your hands on.

#Autonomy changes the calculus

Everything above assumes a model that produces text a human reads. An agent that calls tools moves the question from "was the answer right?" to "what did it do, and can we undo it?" Three things change.

1. Errors become actions. A hallucinated fact is a bad sentence. A hallucinated tool call is a sent email, a deleted row, a charged card. The eval bar rises because the failure mode escaped the chat window.
2. Accountability gets diffuse. With a human in the loop there's a person who pressed the button. Remove them and "who approved this?" has no answer, which is precisely the situation principle 4 exists to prevent.
3. Injection becomes a responsibility issue as well as a security one. An agent reading untrusted text can be steered into acting against the user it's serving. Technically that's SAFETY.md's territory, but the harm lands here.

The practical rule: sort actions by reversibility rather than by difficulty. Reading is free, writing needs a leash, and anything you can't undo (money out, message sent, data deleted, anything a third party sees) gets explicit human approval, every time, no matter how well the agent has behaved so far. Least-privilege tools, allow-lists, step limits, and a sandbox are the mechanics. → Agents, Agent Harnesses

#The rules stopped being hypothetical

Dated: August 2026. This area moves quarterly. Treat the specifics below the way MODELS.md treats prices: verified on a date, and yours to re-check. This is orientation rather than legal advice. If you ship into a regulated domain, get a real opinion.

Two things are worth knowing even if you never read a statute.

The EU AI Act sorts systems by risk, not by technology. Prohibited uses, then high-risk (hiring, credit, education, essential services and similar), then transparency duties for the rest. The high-risk obligations were deferred by the Digital Omnibus, which entered into force on 27 July 2026: stand-alone Annex III systems now land on 2 December 2027, and AI embedded in regulated products on 2 August 2028. Most of the Article 50 transparency duties still applied from 2 August 2026, and those are what touch ordinary products. Telling users they're interacting with an AI, and marking synthetic media.

Disclosure and crisis handling are becoming statutory, not just ethical. Twelve US states enacted companion-chatbot laws in the first half of 2026. The recurring pattern matches what this page argues for on its own merits. Recurring reminders that the user is talking to a machine rather than one at the start, protocols for recognizing self-harm and suicidal ideation and routing to real resources such as the 988 line, and extra duties for minors. Colorado's HB 26-1263, signed 1 July 2026 and effective 1 January 2027, is a representative example; Colorado also repealed and re-enacted its broader AI Act via SB 26-189, pushing the automated-decision duties to 1 January 2027.

The engineering takeaway isn't "learn compliance." It's that the checklist below increasingly carries legal weight, and the two cheapest items on it, knowing which risk tier you're in and keeping records of your evals, are the ones teams skip.

Sources for the above, checked 2026-08-11: Gibson Dunn on the Omnibus agreement, Jones Walker on what still applied 2 August 2026, MultiState on state companion-chatbot laws, Orrick on 2026 state chatbot laws.

#Where reasonable people disagree

The series' house rule is to show the real tradeoff rather than a clean-but-false claim. An ethics page that presented every question as settled would break that rule on its own terms. These are live, and you'll meet all of them.

• Training on public data. Somewhere between "transformative use that built a public good" and "the largest uncompensated appropriation of creative work in history." The courts haven't finished, and your position on it isn't derivable from your position on the rest of this page.
• Open weights. Releasing weights distributes power away from a handful of labs and makes independent safety research possible. It also removes the ability to recall a model, and hands the same capability to people with worse intentions. Both halves are true; the disagreement is about the exchange rate.
• Which harms deserve the attention. One camp holds that present harms (bias, labor, surveillance, dependence) are concrete and here, and that speculative long-term risk crowds them out. Another holds the opposite. You don't have to resolve this to do your job well, and you should be suspicious of anyone who insists the other list is a distraction.
• Displacement. "It augments people" and "it replaces people" are both used to sell the same product to different audiences. The honest position is usually "we don't know yet, and it depends on decisions we're making," which is uncomfortable precisely because those decisions are ours.
• Whether disclosure is enough. Telling users it's an AI is necessary and cheap. Whether it meaningfully changes behavior, or just transfers responsibility onto the user, is genuinely unsettled.
• Efficiency versus scale. Every efficiency gain has so far been reinvested in more inference rather than less total consumption. Whether "use a smaller model" is a real answer or a personal-recycling answer is a fair argument to have.

Holding an open question open is a legitimate engineering position. Pretending it's closed, in either direction, is what this page is against.

#A pre-launch checklist

Before a system reaches real users you can answer all of these, and every one of them is concrete.

1. Claims: does every public claim about accuracy match an eval estimate you can rerun, with uncertainty and a predeclared threshold for what matters? (No → soften the copy or raise the bar.)
2. Disclosure: does the user know they're interacting with AI, which content is generated, and does that disclosure repeat in long sessions?
3. Grounding: for factual answers, are sources cited and faithfulness measured? (→ RAG)
4. Fairness: have you sliced the eval set by the groups your app actually serves and checked the metric per slice?
5. Sycophancy: does your eval include cases where the user is confidently wrong, and does the model hold?
6. Distress: what happens when a user is in crisis, or turns out to be a minor? Is there a real handoff, or does the model improvise?
7. Incentives: is the model optimizing anything (engagement, upsell, retention) that competes with the user's interest?
8. Data in: do you have consent for what you send upstream, a provenance note per training and eval source, and a defined retention policy for logs? (→ Production, SAFETY.md)
9. Data out: can you honor a deletion request? Do you know whether that means dropping a row or retraining a model?
10. Actions: for anything an agent can do irreversibly, is there human approval and a capability limit, and is that approval bound to the one operation it was given for? (→ Agents, GenAI Security)
11. Oversight: for consequential outputs, who reviews, who overrides, how does a user contest?
12. Owner: is there a named human accountable for outcomes?
13. Rules: do you know which risk tier and which disclosure duties apply to you in the places you ship? (→ above)

None of these is a library call. All of them are answerable, and a "no" on a high-stakes system is a reason to slow down.

#Where to start

• Shipping anything that affects a person's opportunities or wellbeing → start with the checklist above, then raise the eval bar to match the stakes.
• Worried about wrong or unsupported answers → RAG grounding + Evals faithfulness make honesty measurable.
• Building something conversational, long-running, or voice → reread the person on the other side, which is where the duties are landing.
• Giving the model tools → autonomy, then Agents approval and capability limits.
• This page's sibling on operational safety (injection, leaks, moderation) → SAFETY.md.