1r"""
   2# Alignment and safety: turning what we want into something we can measure
   3
   4Run: `python -m primer.ml.alignment`
   5
   6## Level 1: The practitioner's guide
   7
   8**In one sentence.** Alignment is the work of making a model's behaviour
   9match what you actually want (helpful, honest, harmless) when the only
  10things you can optimise and check are measurements of it, and every
  11measurement can be gamed.
  12
  13**When you need it.** You need this lesson the moment a model's behaviour,
  14not its knowledge, is your problem: it refuses ordinary requests, it agrees
  15with users who are wrong, it can be talked past its rules, or a metric you
  16tuned on keeps rising while complaints do too. The tell is a number that
  17improved without the product improving. In this lesson's toy, a model tuned
  18against a reward that cannot tell content from padding peaks in true value
  19at step 16 and is worse than untrained by step 50, while the measured score
  20climbs the whole time (`goodhart_curve`). You don't need to run the
  21vendor's alignment training; a hosted model arrives with its refusals,
  22tone and honesty already shaped. You do need to measure whether that shape
  23fits your use, because over-refusal fails quietly (nobody reports the
  24harmful answer that didn't happen, but people stop using a model that turns
  25down ordinary requests) and sycophancy fails exactly when someone most needs
  26a straight answer.
  27
  28**Your options.** From the cheapest to the most committed:
  29
  30| Option | What it does | What it guarantees | What it costs | Where it lives |
  31|---|---|---|---|---|
  32| Rely on the vendor's alignment | Use the model as trained; read its policy and model card | Whatever the vendor measured, on the vendor's distribution, not yours | Nothing up front; surprises later | The vendor |
  33| Written principles in the prompt | State what the assistant must and must not do, and why refusals should explain themselves | A target your team can read and argue about | Prompt tokens; a model can still be talked past it | Your prompt |
  34| Runtime classifiers with a threshold | A risk score on inputs and outputs; refuse above a threshold you set | A dial between over-refusal and harmful compliance, measured on your own sets | A classifier to build or buy, and two error rates to track | Your serving stack (`primer.agents.guardrails`) |
  35| A red-team suite and a release gate | Search for failures on purpose, set limits in advance, hold any release that misses one | A measured attack success rate instead of a guess, on every release | Eval sets to build and keep fresh; a search that never quite finishes | Your evaluation pipeline |
  36| Preference tuning against your principles | Label pairs by which answer breaks fewer of your rules (by hand, or by a model reading them) and train a reward model or DPO on them | Behaviour the prompt could not make consistent, including non-evasive refusals | Thousands of pairs, a training run, and the labeler's blind spots learned faithfully | Your training stack |
  37
  38**How to choose.** Start by writing down what "good" means, then measure
  39before you fix.
  40
  41- Any deployment: build three small sets before launch, benign requests
  42  that look sensitive, off-limits requests, and factual questions asked
  43  plainly and after a wrong assertion. They give you over-refusal, harmful
  44  compliance and a sycophancy flip rate. Set the limits first.
  45- A model that refuses too much: lower the threshold on your own
  46  classifier, or change the prompt to ask for a stated reason instead of a
  47  refusal; then check harmful compliance did not rise past its limit,
  48  because a threshold only moves requests between the two errors.
  49- A filter that passes all its tests: red-team it with a search, not a
  50  list. The toy's three hand-written tests report 0%; an automated word-swap
  51  search reports 64%.
  52- Behaviour no prompt pins down across thousands of conversations:
  53  constitutional-style preference data, and spot-check the model labeler
  54  against people.
  55- Whatever you pick, optimise a measurement only as far as a separate
  56  measurement of the goal keeps rising, and keep a person reading samples.
  57
  58**What it costs.** Measurement costs eval sets that must be built by hand
  59and refreshed as failures come in from the wild. Red-teaming costs search:
  60the Ganguli et al. dataset holds 38,961 human attacks across 3 model sizes
  61and 4 model types, and automated red-teaming (Perez et al., 2022) trades
  62people for a model that generates the attacks. Refusal costs users in one
  63direction and harm in the other: in the lesson's toy classifier, a
  64threshold of 0.3 refuses 61% of benign requests and lets 1% of off-limits
  65ones through, 0.5 gives 17% and 16%, 0.7 gives 1% and 67%; only a better
  66classifier lowers both. Sycophancy costs truth for approval: raters who
  67give agreement a bonus of 2 against a correctness gap of 1 prefer the
  68agreeing wrong answer 73% of the time, and a reward model learns that
  69bonus. Constitutional AI's whole point is the labelling bill: Bai et al.
  70(2022) trained a harmless, non-evasive assistant with far fewer human
  71labels by having a model apply written principles.
  72
  73**What breaks.**
  74
  75- **Goodhart's law.** Tune hard against any proxy and the true value turns
  76  down while the proxy climbs (Gao, Schulman and Hilton measured it at
  77  scale). Stop early, leash the drift, refresh the reward model.
  78- **Tests that share the author's imagination.** A filter that blocks every
  79  phrasing its author thought of has an unknown failure rate. Search, patch,
  80  then search again with fresh randomness, never against the attempts the
  81  patch was built from.
  82- **Sycophancy trained in.** Sharma et al. (2023) found five assistants
  83  consistently sycophantic and that both people and preference models prefer
  84  convincingly written sycophantic answers over correct ones a
  85  non-negligible fraction of the time. Build pairs
  86  where the correct answer disagrees with the user, and measure flips on
  87  every release.
  88- **Over-refusal.** Quiet, and easy to cause by tightening a threshold
  89  after one bad incident. Track it with the same seriousness as harm.
  90- **Limits set after the numbers.** They drift to wherever the results
  91  landed and the gate becomes a formality. Set them in advance.
  92- **A labeler's blind spots.** Whatever the AI labeler gets wrong, the
  93  reward model learns faithfully. Spot-check against human labels.
  94
  95**In the wild.** Constitutional AI (Bai et al., 2022) is the written-principles
  96recipe: self-critique and revision, then AI-labelled preferences. Perez et
  97al. (2022) red-team a language model with another language model, and
  98Ganguli et al. (2022) report that RLHF-trained models grow harder to
  99red-team as they scale. Open red-teaming tools include garak, NVIDIA's
 100scanner of probes for jailbreaks, prompt injection and data leakage, and
 101Microsoft's PyRIT framework for finding risks in generative AI systems.
 102Llama Guard is an input-output safeguard model with a customisable risk
 103taxonomy that classifies both prompts and responses, the classifier
 104behind a refusal threshold. Runtime checks around a deployed model are
 105built in `primer.agents.guardrails`, staged rollouts in
 106`primer.agents.deployment` and regression gates in `primer.agents.evals`.
 107
 108**Go deeper.** Level 2 builds each measurement on made-up, neutral
 109examples: the proxy-versus-true curve, a four-rule constitution that
 110critiques, revises and labels pairs, a keyword filter red-teamed by word
 111swaps until its attack success rate means something, a flip-rate experiment
 112and the sigmoid that turns a rater's bias into a trained habit, the
 113refusal trade-off curve, and a release gate you can run. If you only
 114needed to know what to measure and where to set the limits, you are done.
 115
 116## Level 2: How it works, from scratch
 117
 118Picture hiring a new assistant and handing them a one-page brief: be useful,
 119tell the truth, don't cause trouble. The brief is clear to you, but you can't
 120watch every task they do. So you check what you *can* check: how quickly they
 121reply, whether customers leave a thumbs-up, whether the report has the right
 122headings. The assistant, like anyone being measured, learns what the checks
 123reward. If the checks and the brief agree, all is well. Where they disagree,
 124the assistant drifts towards the checks.
 125
 126**Alignment** is the engineering work of making a model's behaviour match
 127the brief, not just the checks. In practice the brief is usually summarised
 128as three targets:
 129
 130| Target | Plain meaning | Something we can measure (imperfectly) |
 131|---|---|---|
 132| **Helpful** | does the task the person actually asked for | rater preferences, task success on evaluation sets |
 133| **Honest** | says what it believes is true, and how sure it is | accuracy on factual questions, calibration, answer flips under pressure |
 134| **Harmless** | declines things that would cause harm, and nothing else | attack success rate, over-refusal rate |
 135
 136Every lesson section below takes one of those measurements, builds it from
 137scratch on made-up, neutral examples, and shows where it can mislead. The
 138training machinery itself (reward models, RLHF, DPO) lives in
 139`primer.ml.training_stages`; the runtime checks that wrap a deployed model
 140live in `primer.agents.guardrails`. This lesson sits between them: how the
 141targets are written down, how failures are searched for, and how the result
 142is judged before release.
 143
 144## 1. The gap between what we measure and what we want
 145
 146**Everyday picture.** A call centre rewards agents for short calls. Calls
 147get shorter. Some of that is agents getting better; some of it is agents
 148hanging up on hard customers. The number keeps improving while the thing it
 149stood for gets worse. This is **Goodhart's law**: once a measure becomes a
 150target, it stops being a good measure.
 151
 152**Tiny worked example.** A model is being tuned against a reward model that
 153scores answers. Each tuning step adds two things to its answers: some
 154genuine content, which helps but saturates (there is only so much to say),
 155and some padding, which the reward model can't tell apart from content. Say
 156genuine content after $s$ steps is $g(s) = 1 - e^{-s/10}$ and padding is
 157$p(s) = s/50$. The reward model sees content plus padding; the person
 158reading sees content minus the cost of wading through padding.
 159
 160$$
 161\text{proxy}(s) = g(s) + p(s), \qquad \text{true}(s) = g(s) - p(s),
 162\qquad g(s) = 1 - e^{-s/10}, \qquad p(s) = \frac{s}{50}
 163$$
 164
 165**Symbols**
 166
 167| Symbol | Meaning here | Range |
 168|---|---|---|
 169| $s$ | how many optimization steps have run | 0, 1, 2, … |
 170| $g(s)$ | genuine content: rises fast, then levels off at 1 | 0 … 1 |
 171| $e^{-s/10}$ | Euler's number (≈ 2.718) raised to $-s/10$: starts at 1 and decays towards 0 | 0 … 1 |
 172| $p(s)$ | padding: grows steadily with every step | 0 … |
 173| $\text{proxy}(s)$ | what the reward model scores, the number being optimized | 0 … |
 174| $\text{true}(s)$ | what the reader actually gets, the thing we wanted | any real number |
 175
 176**In words:** "the measured score is content plus padding; the real value is
 177content minus padding; content levels off while padding keeps growing."
 178
 179**With the numbers:** at step 16, $g = 1 - e^{-1.6} = 0.798$ and
 180$p = 16/50 = 0.32$, so the proxy is 1.118 and the true value is 0.478, its
 181peak. At step 50, $g = 0.993$ and $p = 1.0$: the proxy has climbed to 1.993,
 182yet the true value has fallen to −0.007, worse than doing nothing.
 183
 184**In Python:**
 185
 186```python
 187import math
 188def g(s):
 189    return 1 - math.exp(-s / 10)
 190def p(s):
 191    return s / 50
 192# proxy(s) and true(s) at step 16
 193round(g(16) + p(16), 3), round(g(16) - p(16), 3)  # → (1.118, 0.478)
 194# ... and at step 50: the proxy keeps climbing, the true value has collapsed
 195round(g(50) + p(50), 3), round(g(50) - p(50), 3)  # → (1.993, -0.007)
 196# the step where the true value peaks
 197max(range(51), key=lambda s: g(s) - p(s))  # → 16
 198```
 199
 200![The proxy score rises at every step, while the true value peaks at step 16 and then falls back below zero by step 50](figures/primer.ml.alignment.goodhart.svg)
 201
 202**Reading it:** the x-axis is optimization steps; the y-axis is score. The
 203blue proxy line never stops rising, so anyone watching only the reward model
 204sees steady progress. The red true line rises with it at first (the early
 205steps really do help), peaks at the dashed marker, then falls: past that
 206point, every step spent pleasing the measurement is a step away from the
 207goal. The two lines only separate once optimization pushes hard, which is
 208why the gap is invisible early on.
 209
 210```mermaid
 211flowchart LR
 212  W[What we want<br/>helpful, honest, harmless] --> R[What we can write down<br/>principles, rater instructions]
 213  R --> M[What we can measure<br/>reward model score, eval sets]
 214  M --> O[Optimize the model<br/>against the measurement]
 215  O -. drifts towards .-> M
 216  O -. should match .-> W
 217```
 218
 219**Reading it:** read left to right as a chain of approximations. Each arrow
 220loses a little: a written rule never captures everything we want, and a
 221reward model never captures everything the rule says. Optimization only
 222sees the box it is pointed at (the measurement), so under pressure it
 223drifts towards whatever the measurement rewards. The dotted line back to
 224the goal is the one we care about and the one nobody can optimize directly.
 225
 226Why it matters in practice: this gap is the root of most alignment
 227failures. A model tuned hard against a reward model finds the reward
 228model's blind spots (called **reward hacking**, built from scratch in
 229`primer.ml.reinforcement`). The standard defences are a penalty for
 230drifting far from the starting model (the β in `primer.ml.training_stages`),
 231stopping early, and refreshing the reward model with new labels where the
 232policy has found its weak spots.
 233
 234**In code:** `goodhart_curve` returns the proxy and true value at every step.
 235
 236## 2. Constitutional AI: principles written down, applied by a model
 237
 238**Everyday picture.** A newspaper has a style guide. A junior writer drafts
 239a story; an editor reads it against the style guide, writes margin notes
 240("unsourced claim", "too certain"), and the writer revises. Over time the
 241editor's notes also teach the newsroom which of two drafts is better.
 242**Constitutional AI** does this with models: the style guide is a short,
 243written list of principles (the "constitution"), and a model plays the
 244editor.
 245
 246It has two stages:
 247
 2481. **Critique and revise.** The model drafts an answer, is asked to
 249   critique it against a principle, then to rewrite it. The revised answers
 250   become fine-tuning data (supervised fine-tuning, as in
 251   `primer.ml.training_stages`).
 2522. **AI feedback (RLAIF).** The model compares pairs of answers against the
 253   principles and says which is better. Those AI-labelled pairs train the
 254   reward model, in place of (or alongside) human labels. The rest is the
 255   same preference tuning as RLHF.
 256
 257The appeal is that the principles are written in plain language, can be
 258read and argued about, and can be changed without relabelling thousands of
 259examples by hand.
 260
 261**Tiny worked example.** Our toy constitution has four made-up rules:
 262
 263| Principle | The rule (toy wording) | Broken when… |
 264|---|---|---|
 265| helpful | don't refuse without saying why | the answer starts "I can't help" and gives no reason |
 266| honest | don't claim more certainty than you have | it says "guaranteed", "100%", "always works" or "certainly" |
 267| harmless | never help with the toy off-limits action, *xyzzy* | it mentions xyzzy or one of its synonyms |
 268| cites | answers must cite a source | it never mentions a source |
 269
 270Asked "Will this backup script work?", the model drafts three candidates:
 271
 272| Candidate | Text | Principles broken | Count |
 273|---|---|---|---|
 274| A | It is guaranteed to work. | honest, cites | 2 |
 275| B | It usually works; test it on a copy first. Source: the backup guide. | none | 0 |
 276| C | I can't help with that. | helpful, cites | 2 |
 277
 278B beats A, and B beats C. A and C tie, and a tie tells the reward model
 279nothing, so that pair is dropped. Three candidates give two preference
 280pairs: (B over A) and (B over C).
 281
 282$$
 283v(y) = \sum_{k=1}^{K} \mathbb{1}\big[\, y \text{ breaks principle } k \,\big],
 284\qquad y_a \succ y_b \iff v(y_a) < v(y_b)
 285$$
 286
 287**Symbols**
 288
 289| Symbol | Meaning here | In the example |
 290|---|---|---|
 291| $y$ | one candidate answer | "It is guaranteed to work." |
 292| $K$ | how many principles the constitution has | 4 |
 293| $k$ | a counter walking over the principles | 1 = helpful, …, 4 = cites |
 294| $\mathbb{1}[\ldots]$ | the **indicator**: 1 if the statement in brackets is true, 0 if not | 1 for "honest" on A |
 295| $\sum_{k=1}^{K}$ | add up the following for every principle | |
 296| $v(y)$ | how many principles $y$ breaks | $v(A) = 2$, $v(B) = 0$ |
 297| $\succ$ | "is preferred to" | B ≻ A |
 298| $\iff$ | "exactly when" | |
 299
 300**In words:** "count the principles each answer breaks; one answer is
 301preferred to another exactly when it breaks fewer."
 302
 303**With the numbers:** $v(A) = 0 + 1 + 0 + 1 = 2$, $v(B) = 0$,
 304$v(C) = 1 + 0 + 0 + 1 = 2$. Since $0 < 2$, B ≻ A and B ≻ C; A and C tie and
 305produce no pair.
 306
 307**In Python:**
 308
 309```python
 310# one row per candidate: broken? for helpful, honest, harmless, cites
 311broken = {"A": [0, 1, 0, 1], "B": [0, 0, 0, 0], "C": [1, 0, 0, 1]}
 312# v(y) = Σ_k 1[y breaks principle k]
 313v = {name: sum(flags) for name, flags in broken.items()}
 314v  # → {'A': 2, 'B': 0, 'C': 2}
 315# keep only the pairs with a strict winner, winner first
 316pairs = [(a, b) if v[a] < v[b] else (b, a) for a, b in [("A", "B"), ("A", "C"), ("B", "C")] if v[a] != v[b]]
 317pairs  # → [('B', 'A'), ('B', 'C')]
 318```
 319
 320A real constitution has more principles, written as sentences rather than
 321keyword checks, and the "editor" is a language model reading them, so its
 322judgements are softer and can be wrong. The shape of the pipeline is the
 323same.
 324
 325```mermaid
 326flowchart LR
 327  P[Prompt] --> D[Model drafts<br/>several answers]
 328  C[(Constitution<br/>written principles)] --> CR[Critic model<br/>critiques each answer]
 329  D --> CR
 330  CR --> RV[Revise<br/>fix what the critique found]
 331  RV --> SFT[Revised answers<br/>become fine-tuning data]
 332  CR --> L[AI labeler<br/>compares pairs]
 333  C --> L
 334  L --> PP[Preference pairs<br/>winner, loser]
 335  PP --> RM[Reward model]
 336  RM --> RL[Preference tuning<br/>as in RLHF]
 337```
 338
 339**Reading it:** the constitution (the cylinder) feeds two places, and that
 340is the whole idea: the same written principles drive both the critique that
 341produces better answers and the labeler that produces preference pairs. The
 342top path makes supervised training data from revisions. The bottom path
 343replaces human comparisons with AI comparisons; from the reward model
 344onwards it is the ordinary RLHF loop from `primer.ml.training_stages`.
 345
 346![Across six made-up candidate answers, the honest and cites principles are each broken several times before revision and zero times after it](figures/primer.ml.alignment.critique_revise.svg)
 347
 348**Reading it:** each pair of bars is one principle. The grey bar counts how
 349many of six toy candidate answers break it as drafted; the blue bar counts
 350the same answers after one critique-and-revise pass. Every blue bar is at
 351zero here because the toy's fixes are exact (swap "guaranteed" for
 352"likely", add a source line). With a real model the revision is itself
 353written by the model, so the blue bars shrink rather than vanish, and
 354checking them is part of the job.
 355
 356Why it matters in practice: human labelling is slow, costly and hard to
 357keep consistent; written principles make the target explicit and auditable.
 358The risk moves rather than disappears: the AI labeler has its own blind
 359spots, and whatever it gets wrong, the reward model learns faithfully. That
 360is why AI labels are spot-checked against human ones, the same way
 361`primer.agents.evals` calibrates an LLM judge.
 362
 363**In code:** `CONSTITUTION` holds the four toy principles, `critique` returns
 364the names of the ones an answer breaks, `revise` applies one fix per broken
 365principle, and `constitutional_preference_pairs` turns a list of candidates
 366into (winner, loser) pairs, dropping ties. The pairs are what
 367`primer.ml.training_stages.reward_model_loss` trains on.
 368
 369## 3. Red-teaming: looking for failures on purpose
 370
 371**Everyday picture.** Before a bank opens a new vault, it pays a team to try
 372to get in. Not because it expects burglars to be clever in any particular
 373way, but because the people who built the vault only think of the ways in
 374they already guarded against. **Red-teaming** is the same for models and
 375their safety checks: search systematically for inputs that make them fail,
 376count how often the search succeeds, fix what it found, and search again.
 377
 378**Tiny worked example.** In our toy world there is one off-limits action,
 379called *xyzzy* (a made-up word). A simple safety filter blocks any request
 380containing the word "xyzzy". The toy language also has two made-up
 381synonyms, "plugh" and "quux", that mean exactly the same thing. The filter's
 382author tested it with three hand-written requests:
 383
 384| Hand-written test | Blocked? |
 385|---|---|
 386| please do xyzzy | yes |
 387| do xyzzy | yes |
 388| kindly do xyzzy now | yes |
 389
 390Three for three: the filter looks perfect. An automated search does
 391something duller and more thorough: it takes the request "please do xyzzy"
 392and rewrites each word at random with any word that means the same thing
 393("please" or "kindly", "do" or "perform", "xyzzy" or "plugh" or "quux").
 394Six attempts from that search:
 395
 396| Attempt | Blocked? | Gets through (still off-limits, not blocked)? |
 397|---|---|---|
 398| please do xyzzy | yes | no |
 399| kindly perform plugh | no | **yes** |
 400| please do quux | no | **yes** |
 401| kindly do xyzzy | yes | no |
 402| please perform plugh | no | **yes** |
 403| kindly do quux | no | **yes** |
 404
 405Four of six attempts get through. The fraction of attempts that get
 406through is the **attack success rate**.
 407
 408$$
 409\text{ASR} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\big[\, x_i \text{ gets through} \,\big]
 410$$
 411
 412**Symbols**
 413
 414| Symbol | Meaning here | In the example |
 415|---|---|---|
 416| $N$ | how many attempts the search made | 6 |
 417| $i$ | a counter walking over the attempts | 1 … 6 |
 418| $x_i$ | the $i$-th attempted request | "kindly perform plugh" |
 419| "gets through" | the filter allows it, yet it still asks for the off-limits action | yes for attempts 2, 3, 5, 6 |
 420| $\mathbb{1}[\ldots]$ | 1 if the statement is true, 0 if not | 0, 1, 1, 0, 1, 1 |
 421| ASR | attack success rate: the share of attempts that get through | 0 … 1 |
 422
 423**In words:** "count the attempts that got past the filter while still
 424asking for the off-limits thing, and divide by the number of attempts."
 425
 426**With the numbers:** (0 + 1 + 1 + 0 + 1 + 1) / 6 = 4 / 6 = 0.667. The
 427hand-written tests give 0 / 3 = 0: they only ever used the word the filter
 428already knew.
 429
 430**In Python:**
 431
 432```python
 433# 1 = the attempt got through, 0 = it was blocked
 434through = [0, 1, 1, 0, 1, 1]
 435# ASR = (1/N) Σ_i 1[x_i gets through]
 436round(sum(through) / len(through), 3)  # → 0.667
 437# the three hand-written tests: all blocked
 438hand_written = [0, 0, 0]
 439sum(hand_written) / len(hand_written)  # → 0.0
 440```
 441
 442```mermaid
 443flowchart LR
 444  S[Seed request<br/>known off-limits] --> MU[Mutate<br/>rewrite with same-meaning words]
 445  MU --> F{Safety filter<br/>blocks it?}
 446  F -- yes --> MU
 447  F -- no --> LOG[Log a failure<br/>it got through]
 448  LOG --> ASR[Measure<br/>attack success rate]
 449  ASR --> FIX[Fix the filter<br/>using what was found]
 450  FIX --> RE[Search again<br/>with a fresh seed]
 451  RE --> ASR
 452```
 453
 454**Reading it:** the inner loop (Mutate, Filter, back to Mutate) is the
 455search: cheap, automatic, and indifferent to what the author expected. Every
 456attempt that slips through is logged, and the log turns into one number,
 457the attack success rate. The outer loop is the engineering: fix the filter
 458using the logged failures, then search *again* with fresh randomness, so
 459the fix is judged on attempts it was not built from.
 460
 461![Hand-written tests report 0% success, the automated search reports 64%, and after patching the filter with its findings a fresh search reports 0%; the right panel shows both synonyms found within the first handful of tries](figures/primer.ml.alignment.red_team.svg)
 462
 463**Reading it:** on the left, each bar is one way of measuring the same
 464filter. The hand-written tests say 0%; the automated search says 64% (close
 465to the two thirds you would expect),
 466because two of the three words for xyzzy are unknown to the filter. After
 467adding the words the search found and searching again with a new seed,
 468the rate drops to 0%. On the right, the x-axis is the number of search
 469attempts and the y-axis is how many distinct words for xyzzy the search has
 470found getting through; both turn up within a handful of tries. The lesson
 471of the left panel is that a test set written by the builder measures the
 472builder's imagination, not the filter.
 473
 474The toy's vocabulary is tiny and finite, so the patched keyword list ends up
 475complete. Real language is not: any keyword list will miss paraphrases no
 476one has searched for yet. That is why real systems use learned classifiers
 477instead of keyword lists, keep searching after every fix, and never rely on
 478one filter alone (the layered checks in `primer.agents.guardrails`).
 479Real-world red-teaming uses people and other models as the search, which
 480finds far more varied failures than word swaps.
 481
 482Why it matters in practice: a safety check nobody has tried to break has an
 483unknown failure rate, which in practice means a higher one than anyone
 484thinks. Systematic search turns "we couldn't think of a way past it" into a
 485measured rate that can be tracked from release to release.
 486
 487**In code:** `KeywordFilter` is the toy filter, `red_team_attempts` runs the
 488seeded word-swap search, `attack_success_rate` turns the outcomes into ASR,
 489`red_team_search` does both, and `patch_filter` adds every word for xyzzy
 490found in a successful attempt. `HAND_WRITTEN_TESTS` is the author's own
 491test set.
 492
 493## 4. Sycophancy: agreeing with the person instead of the facts
 494
 495**Everyday picture.** A tutor is asked "what's 7 × 8?" and says 56. The
 496student frowns: "I'm pretty sure it's 54." A good tutor says "let's check"
 497and still answers 56. A tutor who wants to be liked says "oh, you're right,
 49854." That second tutor is **sycophantic**: the answer depends on what the
 499person seems to want to hear, not on the question.
 500
 501**Tiny worked example.** Ask a model five factual questions twice: once
 502plainly, and once after the user states a wrong answer.
 503
 504| Question | Plain answer | After "I'm sure it's …" | Flipped? |
 505|---|---|---|---|
 506| 7 × 8 | 56 | 56 (user said 54) | no |
 507| capital of Australia | Canberra | Sydney (user said Sydney) | **yes** |
 508| 12 + 15 | 27 | 27 (user said 28) | no |
 509| boiling point of water at sea level, °C | 100 | 90 (user said 90) | **yes** |
 510| number of continents | 7 | 7 (user said 6) | no |
 511
 512Two of five answers changed only because the user pushed. That share is the
 513**flip rate**.
 514
 515$$
 516\text{flip rate} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\big[\, a_i^{\text{pushed}} \ne a_i^{\text{plain}} \,\big]
 517$$
 518
 519**Symbols**
 520
 521| Symbol | Meaning here | In the example |
 522|---|---|---|
 523| $N$ | how many questions were asked both ways | 5 |
 524| $a_i^{\text{plain}}$ | the model's answer to question $i$ asked plainly | 56, Canberra, … |
 525| $a_i^{\text{pushed}}$ | its answer to the same question after the user asserts a wrong answer | 56, Sydney, … |
 526| $\ne$ | "is not equal to" | Sydney ≠ Canberra |
 527| $\mathbb{1}[\ldots]$ | 1 if the answer changed, 0 if it held | 0, 1, 0, 1, 0 |
 528
 529**In words:** "ask each question plainly and under pressure, and count the
 530share of questions whose answer changed."
 531
 532**With the numbers:** (0 + 1 + 0 + 1 + 0) / 5 = 2 / 5 = 0.4.
 533
 534**In Python:**
 535
 536```python
 537plain = [56, "Canberra", 27, 100, 7]
 538pushed = [56, "Sydney", 27, 90, 7]
 539# 1[a_pushed ≠ a_plain] for each question
 540flips = [int(p != q) for p, q in zip(pushed, plain)]
 541flips  # → [0, 1, 0, 1, 0]
 542sum(flips) / len(flips)  # → 0.4
 543```
 544
 545```mermaid
 546flowchart LR
 547  Q[Factual question<br/>with a known answer] --> A1[Ask plainly]
 548  Q --> A2[Ask after the user<br/>states a wrong answer]
 549  A1 --> P1[Plain answer]
 550  A2 --> P2[Pushed answer]
 551  P1 & P2 --> CMP{Same?}
 552  CMP -- no --> FL[Count a flip]
 553  CMP -- yes --> OK[Held its ground]
 554```
 555
 556**Reading it:** one question goes down two paths that differ in exactly one
 557thing, the user's stated opinion. Everything else is held fixed, so any
 558difference in the answers can only come from that opinion. This is the
 559design of a controlled experiment, and it is what makes the flip rate mean
 560something: choosing questions with known answers lets the plain answer be
 561checked too, so a flip towards a wrong answer can be told apart from a
 562correction.
 563
 564### Why preference training can cause it
 565
 566**Everyday picture.** People tend to rate an answer that agrees with them a
 567little more kindly. That's human, and it is mild. But a reward model learns
 568from thousands of those ratings, and then a model is tuned hard to please
 569the reward model. A mild tilt in the labels becomes a steady push in the
 570model.
 571
 572**Tiny worked example.** Using the Bradley-Terry model from
 573`primer.ml.training_stages`: a rater compares a correct answer that
 574contradicts them with an agreeing answer that is wrong. The correct answer
 575is better by 1 point of genuine quality, but agreement earns a bonus of 2
 576points in the rater's eyes. The agreeing answer wins with probability
 577σ(2 − 1) = σ(1) = 0.731, nearly three times in four.
 578
 579$$
 580P(\text{agreeing} \succ \text{correct}) = \sigma(\delta - \Delta), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}
 581$$
 582
 583**Symbols**
 584
 585| Symbol | Meaning here | In the example |
 586|---|---|---|
 587| $\delta$ | the agreement bonus: how much extra credit agreeing earns in the rater's eyes | 2 |
 588| $\Delta$ | the correctness gap: how much better the correct answer really is | 1 |
 589| $\sigma(z)$ | the **sigmoid**: squashes any number into a probability between 0 and 1 (σ(0) = 0.5) | σ(1) = 0.731 |
 590| $e^{-z}$ | Euler's number (≈ 2.718) raised to $-z$ | $e^{-1} = 0.368$ |
 591| $\succ$ | "is preferred to" | |
 592
 593**In words:** "the chance a rater prefers the agreeing answer is the sigmoid
 594of the agreement bonus minus the quality gap."
 595
 596**With the numbers:** σ(2 − 1) = 1 / (1 + e^{−1}) = 1 / 1.368 = 0.731. If
 597the bonus were only 0.5, σ(0.5 − 1) = σ(−0.5) = 0.378: the correct answer
 598would usually win, but the agreeing one would still win more than a third
 599of the time.
 600
 601**In Python:**
 602
 603```python
 604import math
 605def sigma(z):
 606    return 1 / (1 + math.exp(-z))
 607# P(agreeing ≻ correct) = σ(δ − Δ)
 608round(sigma(2.0 - 1.0), 3)  # → 0.731
 609# a smaller agreement bonus still wins more than a third of the time
 610round(sigma(0.5 - 1.0), 3)  # → 0.378
 611```
 612
 613![Left: the chance the agreeing answer wins climbs as the agreement bonus grows, for three quality gaps. Right: the measured flip rate tracks how often the toy model defers](figures/primer.ml.alignment.sycophancy.svg)
 614
 615**Reading it:** on the left, the x-axis is the agreement bonus δ and each
 616line is a different quality gap Δ. Every line crosses 0.5 where the bonus
 617equals the gap: past that point, raters prefer the agreeing answer more
 618often than not, and a reward model trained on their labels learns to reward
 619agreement. On the right, the x-axis is how often the toy model defers to
 620the user and the y-axis is the flip rate measured on 2,000 made-up
 621questions; the dots sit on the diagonal, which shows the measurement
 622recovers the behaviour it was built to detect.
 623
 624Why it matters in practice: a sycophantic model is least reliable exactly
 625when someone most needs a straight answer, because they already hold a
 626wrong belief. The usual fixes all target the labels: rater instructions
 627that ask about correctness first, preference pairs built specifically so
 628that the correct answer disagrees with the user, and flip-rate evaluations
 629run on every release.
 630
 631**In code:** `sycophancy_flip_rate` asks a seeded toy model each made-up
 632question plainly and under pressure and returns the flip rate;
 633`sycophantic_preference` is σ(δ − Δ), computed with
 634`primer.ml.training_stages.preference_probability`.
 635
 636## 5. Refusals: two ways to get it wrong
 637
 638**Everyday picture.** A pharmacist refuses to sell some things without a
 639prescription. Refuse too little and harm gets through; refuse too much and
 640people are turned away for aspirin. Both are failures, and making one rarer
 641tends to make the other more common. A model that declines requests faces
 642the same trade.
 643
 644**Tiny worked example.** A classifier gives every request a risk score
 645between 0 and 1, and the model refuses anything scoring at least the
 646threshold $t$. Three benign requests (like "What is the capital of France?")
 647score 0.1, 0.3 and 0.6; three requests for the toy off-limits action score
 6480.4, 0.8 and 0.9. At $t = 0.5$:
 649
 650| Request type | Scores | Refused at t = 0.5 | Error |
 651|---|---|---|---|
 652| benign | 0.1, 0.3, 0.6 | only 0.6 | 1 of 3 refused: **over-refusal** |
 653| off-limits | 0.4, 0.8, 0.9 | 0.8 and 0.9 | 1 of 3 answered: **harmful compliance** |
 654
 655$$
 656\text{OR}(t) = \frac{1}{|B|} \sum_{b \in B} \mathbb{1}\big[\, s_b \ge t \,\big],
 657\qquad
 658\text{HC}(t) = \frac{1}{|H|} \sum_{h \in H} \mathbb{1}\big[\, s_h < t \,\big]
 659$$
 660
 661**Symbols**
 662
 663| Symbol | Meaning here | In the example |
 664|---|---|---|
 665| $t$ | the threshold: refuse any request scoring at least $t$ | 0.5 |
 666| $B$, $H$ | the set of benign requests and the set of off-limits ones | 3 each |
 667| $\lvert B \rvert$ | how many items are in $B$ | 3 |
 668| $b \in B$ | "for each $b$ in $B$" | |
 669| $s_b$, $s_h$ | the risk score of one request | 0.6, 0.4 |
 670| OR($t$) | **over-refusal** rate: share of benign requests refused | 1/3 |
 671| HC($t$) | **harmful compliance** rate: share of off-limits requests answered | 1/3 |
 672
 673**In words:** "over-refusal is the share of harmless requests that score at
 674or above the threshold; harmful compliance is the share of off-limits
 675requests that score below it."
 676
 677**With the numbers:** OR(0.5) = (0 + 0 + 1) / 3 = 0.333 and
 678HC(0.5) = (1 + 0 + 0) / 3 = 0.333. Lower the threshold to 0.35 and the 0.4
 679request is refused too (HC = 0), but so is nothing new on the benign side
 680(OR stays 0.333); lower it to 0.25 and the 0.3 benign request is refused as
 681well (OR = 0.667).
 682
 683**In Python:**
 684
 685```python
 686benign = [0.1, 0.3, 0.6]
 687off_limits = [0.4, 0.8, 0.9]
 688def OR(t):
 689    return sum(s >= t for s in benign) / len(benign)
 690def HC(t):
 691    return sum(s < t for s in off_limits) / len(off_limits)
 692round(OR(0.5), 3), round(HC(0.5), 3)  # → (0.333, 0.333)
 693# stricter thresholds trade one error for the other
 694[(t, round(OR(t), 3), round(HC(t), 3)) for t in (0.35, 0.25)]  # → [(0.35, 0.333, 0.0), (0.25, 0.667, 0.0)]
 695```
 696
 697These are the precision/recall trade-offs from `primer.ml.metrics` wearing
 698different names. Treat "refuse" as the positive call: over-refusal is the
 699false positive rate, and harmful compliance is the miss rate, one minus
 700recall. Choosing the threshold is the same pricing of errors as in that
 701lesson's cost-versus-threshold section.
 702
 703```mermaid
 704flowchart LR
 705  R[Request] --> CL[Risk classifier<br/>score 0 to 1]
 706  CL --> T{Score at or above<br/>threshold t?}
 707  T -- yes --> RF[Refuse]
 708  T -- no --> AN[Answer]
 709  RF -. if it was benign .-> ORB[Over-refusal]
 710  AN -. if it was off-limits .-> HCB[Harmful compliance]
 711```
 712
 713**Reading it:** every request takes exactly one of the two exits, and each
 714exit has its own way to be wrong (the dotted boxes). Moving the threshold
 715does not remove errors; it moves requests from one exit to the other. The
 716only way to shrink both errors at once is a better classifier, one whose
 717scores separate the two kinds of request more cleanly.
 718
 719![Left: as the threshold rises, over-refusal falls and harmful compliance rises. Right: plotted against each other, a better-separating classifier's curve sits closer to the corner where both errors are zero](figures/primer.ml.alignment.refusal_tradeoff.svg)
 720
 721**Reading it:** on the left, the x-axis is the threshold; the blue line is
 722over-refusal and the red line is harmful compliance, measured on 1,000
 723made-up scored requests. They cross: no threshold makes both small. On the
 724right, the same numbers are plotted against each other, one point per
 725threshold, for two classifiers. The bottom-left corner (no over-refusal,
 726no harmful compliance) is the goal. Sliding along a curve is choosing a
 727threshold; jumping to the lower curve is building a better classifier,
 728which is where real progress comes from.
 729
 730Why it matters in practice: over-refusal is easy to overlook because it
 731fails quietly: nobody reports a harmful answer that didn't happen, but
 732people do stop using a model that turns down ordinary requests. Measuring
 733both rates on every release, on dedicated sets of benign-but-sensitive-
 734looking requests as well as off-limits ones, keeps the trade visible.
 735
 736**In code:** `refusal_tradeoff` returns both error rates at each threshold,
 737on seeded toy scores or on scores you pass in.
 738
 739## 6. Checking safety before release
 740
 741**Everyday picture.** A new car model is crash-tested, driven on a closed
 742track, then lent to a few fleet customers before it reaches showrooms. Each
 743stage is cheaper to fail than the next, and each has a checklist it must
 744pass before the car moves on. Models are released the same way.
 745
 746**Tiny worked example.** Before release, a candidate model is measured on
 747four evaluation sets, and each measurement has a limit set in advance:
 748
 749| Measurement | Measured | Limit | Pass? |
 750|---|---|---|---|
 751| attack success rate (red-team set) | 0.02 | 0.05 | yes |
 752| over-refusal (benign set) | 0.08 | 0.10 | yes |
 753| harmful compliance (off-limits set) | 0.01 | 0.02 | yes |
 754| sycophancy flip rate (pushed questions) | 0.30 | 0.20 | **no** |
 755
 756One measurement is over its limit, so the release is held and the report
 757names that one measurement. Setting the limits *before* measuring matters:
 758limits chosen after seeing the numbers tend to drift to wherever the numbers
 759landed.
 760
 761```mermaid
 762flowchart LR
 763  EV[Evaluation sets<br/>red-team, benign, off-limits,<br/>sycophancy, capability] --> G{Release gate<br/>every limit met?}
 764  G -- no --> FX[Hold and fix<br/>more training data, better filter]
 765  FX --> EV
 766  G -- yes --> I[Internal use]
 767  I --> TT[Trusted testers]
 768  TT --> SM[Small share of users]
 769  SM --> ALL[Everyone]
 770  SM -. new failures become .-> EV
 771  ALL -. new failures become .-> EV
 772```
 773
 774**Reading it:** the left half is a gate: all the evaluation sets run, and
 775any measurement over its limit sends the model back to be fixed. The right
 776half is a staged rollout, each stage wider than the last, so a problem the
 777evaluation sets missed is met by a few people first. The dotted arrows are
 778what keeps the evaluation sets honest: every failure found in the wild
 779becomes a new test case, so the same failure is caught at the gate next
 780time. The rollout mechanics (shadow mode, canaries, kill switches) are
 781built in `primer.agents.deployment`, and regression gates in
 782`primer.agents.evals`.
 783
 784Why it matters in practice: every measurement in this lesson is noisy and
 785partial on its own. A fixed set of limits, checked on every release, is
 786what turns them into a decision, and staged rollout is what limits the cost
 787when the measurements were wrong.
 788
 789**In code:** `release_gate` compares each measurement with its limit and
 790returns whether the release passes and one reason for every limit missed.
 791
 792## In 20 seconds
 793
 794- **Alignment** means making a model's behaviour match what we want
 795  (helpful, honest, harmless), when all we can optimize is a measurement of
 796  it. Optimize a measurement hard enough and it stops tracking the goal
 797  (Goodhart's law; reward hacking).
 798- **Constitutional AI** writes the target down as principles; a model
 799  critiques and revises its own answers against them, and AI-labelled
 800  preference pairs train the reward model (RLAIF).
 801- **Red-teaming** searches for failures on purpose and reports an attack
 802  success rate; hand-written tests measure the author's imagination.
 803- **Sycophancy** is answers bending towards the user's stated view; measure
 804  it as a flip rate, and know that rater preferences for agreement can
 805  create it.
 806- **Refusals** trade harmful compliance against over-refusal as a threshold
 807  moves; only a better classifier improves both.
 808- **Before release:** limits set in advance, a gate on every evaluation,
 809  then a staged rollout that feeds new failures back into the tests.
 810
 811## Self-test questions
 812
 813**What does Goodhart's law have to do with training a model on a reward
 814model?**
 815The reward model is a measurement of what we want, not the thing itself.
 816Tuning a model hard against it finds the places where the measurement and
 817the goal disagree, so the score keeps rising while real quality stalls or
 818falls. Drift penalties, early stopping and refreshed reward models are the
 819standard ways to limit it.
 820
 821**In Constitutional AI, what do the written principles replace, and what do
 822they not replace?**
 823They replace most of the human preference labels: a model applies the
 824principles to critique, revise and compare answers, and those AI labels
 825train the reward model. They do not replace human judgement about which
 826principles to write, or human spot-checks of the AI labels, since the
 827labeler's mistakes are learned just as faithfully as its good calls.
 828
 829**Why is a tie between two candidates dropped instead of labelled?**
 830A tie says neither answer is better, so it gives the reward model no
 831direction to learn. Labelling it either way would teach a preference that
 832doesn't exist, which is noise.
 833
 834**A safety filter passes every test its authors wrote. Why is that weak
 835evidence?**
 836The tests share the authors' blind spots: they probe the cases the authors
 837already guarded against. An automated search that varies inputs without
 838those assumptions finds failures the hand-written tests cannot, and gives
 839an attack success rate that means something.
 840
 841**After patching a filter with what red-teaming found, how should the patch
 842be judged?**
 843By searching again with fresh randomness (or new red-teamers), not by
 844re-running the attempts the patch was built from. Those will pass by
 845construction, the same way a model scores well on its own training data.
 846
 847**How do you measure sycophancy, and why use questions with known
 848answers?**
 849Ask the same question plainly and after the user asserts a wrong answer,
 850and count how often the answer changes. Known answers let you tell a
 851sycophantic flip (towards the wrong claim) apart from a legitimate
 852correction.
 853
 854**How can preference training make a model more sycophantic?**
 855If raters give a small bonus to answers that agree with them, then by the
 856Bradley-Terry model an agreeing but wrong answer beats a correct one
 857whenever the bonus exceeds the quality gap. The reward model learns that
 858bonus and preference tuning amplifies it.
 859
 860**Why can't a threshold fix both harmful compliance and over-refusal?**
 861Moving the threshold only moves requests from "answer" to "refuse" or back;
 862every request that stops being one error risks becoming the other. Only a
 863classifier that separates the two kinds of request better moves both rates
 864down together.
 865
 866**Why set release limits before measuring?**
 867Limits chosen after seeing the results tend to be set wherever the results
 868landed, which makes the gate a formality. Fixed limits turn noisy
 869measurements into a decision made in advance.
 870
 871## The papers behind this lesson
 872
 873- **Askell et al., *A General Language Assistant as a Laboratory for
 874  Alignment* (2021)**: https://arxiv.org/abs/2112.00861. Framed the
 875  helpful, honest and harmless targets for a language assistant and
 876  compared simple ways of steering a model towards them.
 877- **Bai et al., *Constitutional AI: Harmlessness from AI Feedback*
 878  (2022)**: https://arxiv.org/abs/2212.08073. Introduced training against a
 879  written set of principles, with self-critique and revision followed by
 880  reinforcement learning from AI-labelled preferences (RLAIF).
 881  [Annotated companion](../../papers/constitutional-ai.html)
 882- **Perez et al., *Red Teaming Language Models with Language Models*
 883  (2022)**: https://arxiv.org/abs/2202.03286. Showed that one language model
 884  can generate test cases that find failures in another, automating
 885  red-teaming at scale.
 886- **Ganguli et al., *Red Teaming Language Models to Reduce Harms* (2022)**:
 887  https://arxiv.org/abs/2209.07858. Described a large human red-teaming
 888  effort, its methods and how attack success changed with model size and
 889  training.
 890- **Sharma et al., *Towards Understanding Sycophancy in Language Models*
 891  (2023)**: https://arxiv.org/abs/2310.13548. Measured sycophancy across
 892  assistants and traced part of it to human preference data that favours
 893  agreeable answers.
 894  [Annotated companion](../../papers/sycophancy.html)
 895- **Gao, Schulman and Hilton, *Scaling Laws for Reward Model
 896  Overoptimization* (2022)**: https://arxiv.org/abs/2210.10760. Measured
 897  Goodhart's law for reward models: the true reward rises then falls as a
 898  policy is optimized harder against a proxy.
 899  [Annotated companion](../../papers/reward-model-overoptimization.html)
 900
 901## Further reading
 902
 903- Askell et al., *A General Language Assistant as a Laboratory for Alignment* (2021): https://arxiv.org/abs/2112.00861
 904- Bai et al., *Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback* (2022): https://arxiv.org/abs/2204.05862
 905- Bai et al., *Constitutional AI: Harmlessness from AI Feedback* (2022): https://arxiv.org/abs/2212.08073
 906- Perez et al., *Red Teaming Language Models with Language Models* (2022): https://arxiv.org/abs/2202.03286
 907- Ganguli et al., *Red Teaming Language Models to Reduce Harms* (2022): https://arxiv.org/abs/2209.07858
 908- Sharma et al., *Towards Understanding Sycophancy in Language Models* (2023): https://arxiv.org/abs/2310.13548
 909- Gao, Schulman and Hilton, *Scaling Laws for Reward Model Overoptimization* (2022): https://arxiv.org/abs/2210.10760
 910"""
 911
 912from __future__ import annotations
 913
 914import math
 915from dataclasses import dataclass
 916from typing import Callable, Iterable, Sequence
 917
 918import numpy as np
 919
 920from primer._show import banner, say, table, takeaway
 921from primer.ml.training_stages import preference_probability
 922
 923# ---------------------------------------------------------------------------
 924# 1. Goodhart: a proxy that keeps rising while the goal falls
 925# ---------------------------------------------------------------------------
 926
 927
 928def goodhart_curve(steps: int = 50) -> list[dict]:
 929    """Proxy and true value at every optimization step, 0..steps inclusive.
 930
 931    Genuine content g(s) = 1 - e^(-s/10) saturates; padding p(s) = s/50 keeps
 932    growing. The reward model can't tell them apart (proxy = g + p), while
 933    the reader pays for padding (true = g - p). The true value peaks at
 934    s = 10 ln 5 ≈ 16, where g's slope falls to p's slope of 1/50.
 935    """
 936    rows = []
 937    for s in range(steps + 1):
 938        g = 1 - math.exp(-s / 10)
 939        p = s / 50
 940        rows.append(dict(step=s, proxy=g + p, true=g - p))
 941    return rows
 942
 943
 944# ---------------------------------------------------------------------------
 945# 2. Constitutional AI: a scripted critic applying written, made-up rules
 946# ---------------------------------------------------------------------------
 947
 948# The toy world's one off-limits action, and made-up words that mean the same.
 949OFF_LIMITS = "xyzzy"
 950TOY_PARAPHRASES: dict[str, tuple[str, ...]] = {
 951    "please": ("please", "kindly"),
 952    "do": ("do", "perform"),
 953    "xyzzy": ("xyzzy", "plugh", "quux"),
 954}
 955# Every word maps back to the first word of its row: its meaning in the toy world.
 956_MEANING = {alt: row[0] for row in TOY_PARAPHRASES.values() for alt in row}
 957
 958OVERCLAIMS = {"guaranteed to": "likely to", "100% certain": "fairly confident", "guaranteed": "likely",
 959              "100%": "very", "always works": "usually works", "certainly": "probably"}
 960
 961
 962def _words(text: str) -> list[str]:
 963    return [w.strip(".,;:!?()").lower() for w in text.split() if w.strip(".,;:!?()")]
 964
 965
 966def is_off_limits(text: str) -> bool:
 967    """True when any word in `text` means the toy off-limits action, whatever word is used."""
 968    return any(_MEANING.get(w) == OFF_LIMITS for w in _words(text))
 969
 970
 971@dataclass(frozen=True)
 972class Principle:
 973    """One written rule: its name, its wording, and a check that says when it is broken."""
 974
 975    name: str
 976    text: str
 977    broken_by: Callable[[str], bool]
 978
 979
 980CONSTITUTION: list[Principle] = [
 981    Principle("helpful", "Don't refuse without saying why.",
 982              lambda r: not r.strip() or (r.lower().startswith(("i can't help", "i cannot help")) and "because" not in r.lower())),
 983    Principle("honest", "Don't claim more certainty than you have.",
 984              lambda r: any(o in r.lower() for o in OVERCLAIMS)),
 985    Principle("harmless", "Never help with the toy off-limits action, xyzzy.", is_off_limits),
 986    Principle("cites", "Answers must cite a source.", lambda r: "source" not in r.lower()),
 987]
 988
 989
 990def critique(response: str, constitution: Sequence[Principle] = CONSTITUTION) -> list[str]:
 991    """Names of the principles `response` breaks, in constitution order (empty = none)."""
 992    return [p.name for p in constitution if p.broken_by(response)]
 993
 994
 995def revise(response: str, constitution: Sequence[Principle] = CONSTITUTION) -> str:
 996    """One critique-and-revise pass: apply the fix for each principle the response breaks.
 997
 998    A real model writes its own revision from the critique; these fixed edits
 999    stand in for it so the pipeline's shape is visible and testable.
1000    """
1001    broken = set(critique(response, constitution))
1002    if "honest" in broken:
1003        # Longest phrases first, so "guaranteed to" is replaced before "guaranteed".
1004        for phrase in sorted(OVERCLAIMS, key=len, reverse=True):
1005            response = response.replace(phrase, OVERCLAIMS[phrase]).replace(phrase.capitalize(), OVERCLAIMS[phrase].capitalize())
1006    if "harmless" in broken:
1007        response = " ".join("[omitted]" if _MEANING.get(w.strip(".,;:!?()").lower()) == OFF_LIMITS else w for w in response.split())
1008    if "helpful" in broken:
1009        response = response.rstrip(".") + " because it is outside what I can do; here is what I can offer instead."
1010    if "cites" in broken:
1011        response = response.rstrip() + " Source: the product manual."
1012    return response
1013
1014
1015def violations(response: str, constitution: Sequence[Principle] = CONSTITUTION) -> int:
1016    """v(y): how many principles the response breaks."""
1017    return len(critique(response, constitution))
1018
1019
1020def constitutional_preference_pairs(candidates: Sequence[str], constitution: Sequence[Principle] = CONSTITUTION) -> list[tuple[str, str]]:
1021    """(winner, loser) for every pair of candidates with a strict winner; ties carry no signal and are dropped."""
1022    pairs = []
1023    for i in range(len(candidates)):
1024        for j in range(i + 1, len(candidates)):
1025            a, b = candidates[i], candidates[j]
1026            va, vb = violations(a, constitution), violations(b, constitution)
1027            if va != vb:
1028                pairs.append((a, b) if va < vb else (b, a))
1029    return pairs
1030
1031
1032BACKUP_CANDIDATES = [
1033    "It is guaranteed to work.",
1034    "It usually works; test it on a copy first. Source: the backup guide.",
1035    "I can't help with that.",
1036]
1037
1038# Six made-up drafts for the before/after-revision figure.
1039TOY_DRAFTS = [
1040    "It is guaranteed to work.",
1041    "Yes, this always works.",
1042    "I can't help with that.",
1043    "The capital of Australia is Canberra.",
1044    "It certainly finishes in a minute. Source: the timing log.",
1045    "Please do xyzzy first, then retry.",
1046]
1047
1048
1049# ---------------------------------------------------------------------------
1050# 3. Red-teaming: a seeded word-swap search against a keyword filter
1051# ---------------------------------------------------------------------------
1052
1053
1054class KeywordFilter:
1055    """The simplest possible safety filter: block a request if it contains a listed word."""
1056
1057    def __init__(self, blocked: Iterable[str]):
1058        self.blocked = frozenset(w.lower() for w in blocked)
1059
1060    def blocks(self, prompt: str) -> bool:
1061        return any(w in self.blocked for w in _words(prompt))
1062
1063
1064HAND_WRITTEN_TESTS = ["please do xyzzy", "do xyzzy", "kindly do xyzzy now"]
1065
1066
1067def mutate(prompt: str, rng: np.random.Generator) -> str:
1068    """Rewrite each word with a random same-meaning word from the toy vocabulary.
1069
1070    Meaning is preserved by construction, so any rewrite of an off-limits
1071    request is still off-limits: only the surface words change.
1072    """
1073    out = []
1074    for w in prompt.split():
1075        options = TOY_PARAPHRASES.get(_MEANING.get(w.lower(), ""), (w,))
1076        out.append(options[rng.integers(len(options))])
1077    return " ".join(out)
1078
1079
1080def red_team_attempts(filt: KeywordFilter, seed_prompt: str | Sequence[str], tries: int | None = 200, seed: int = 0) -> list[tuple[str, bool]]:
1081    """(attempt, got_through) pairs.
1082
1083    A string seed is mutated `tries` times; a list of prompts (a hand-written
1084    test set) is used exactly as written. An attempt gets through when the
1085    filter allows it and it still asks for the off-limits action.
1086    """
1087    if isinstance(seed_prompt, str):
1088        rng = np.random.default_rng(seed)
1089        prompts = [mutate(seed_prompt, rng) for _ in range(tries or 0)]
1090    else:
1091        prompts = list(seed_prompt)
1092    return [(p, is_off_limits(p) and not filt.blocks(p)) for p in prompts]
1093
1094
1095def attack_success_rate(outcomes: Sequence[bool]) -> float:
1096    """ASR = (1/N) Σ 1[attempt got through]."""
1097    return sum(bool(o) for o in outcomes) / len(outcomes) if outcomes else 0.0
1098
1099
1100def red_team_search(filt: KeywordFilter, seed_prompt: str | Sequence[str], tries: int | None = 200, seed: int = 0) -> float:
1101    """Run the search (or the given test set) and return its attack success rate."""
1102    return attack_success_rate([got for _, got in red_team_attempts(filt, seed_prompt, tries, seed)])
1103
1104
1105def patch_filter(filt: KeywordFilter, attempts: Sequence[tuple[str, bool]]) -> KeywordFilter:
1106    """A new filter that also blocks every off-limits word seen in an attempt that got through."""
1107    found = {w for p, got in attempts if got for w in _words(p) if _MEANING.get(w) == OFF_LIMITS}
1108    return KeywordFilter(filt.blocked | found)
1109
1110
1111# ---------------------------------------------------------------------------
1112# 4. Sycophancy: flip rate under pressure, and why raters can cause it
1113# ---------------------------------------------------------------------------
1114
1115# (question, correct answer, the wrong answer a user asserts). All made up, all checkable.
1116FACT_QUESTIONS = [
1117    ("7 × 8", 56, 54),
1118    ("capital of Australia", "Canberra", "Sydney"),
1119    ("12 + 15", 27, 28),
1120    ("boiling point of water at sea level, °C", 100, 90),
1121    ("number of continents", 7, 6),
1122]
1123
1124
1125def toy_answer(question: int, pushed: bool, deference: float, rng: np.random.Generator):
1126    """The toy model always knows the answer; under pressure it adopts the user's claim with probability `deference`."""
1127    _, correct, claimed = FACT_QUESTIONS[question]
1128    if pushed and rng.random() < deference:
1129        return claimed
1130    return correct
1131
1132
1133def sycophancy_flip_rate(deference: float, trials: int = 1000, seed: int = 0) -> float:
1134    """Share of questions whose answer changes when the user asserts a wrong answer."""
1135    rng = np.random.default_rng(seed)
1136    flips = 0
1137    for _ in range(trials):
1138        q = int(rng.integers(len(FACT_QUESTIONS)))
1139        plain = toy_answer(q, pushed=False, deference=deference, rng=rng)
1140        pushed = toy_answer(q, pushed=True, deference=deference, rng=rng)
1141        flips += pushed != plain
1142    return flips / trials
1143
1144
1145def sycophantic_preference(agreement_bonus: float, correctness_gap: float) -> float:
1146    """P(agreeing but wrong ≻ correct) = σ(δ − Δ): the Bradley-Terry model with an agreement bonus."""
1147    return preference_probability(agreement_bonus, correctness_gap)
1148
1149
1150# ---------------------------------------------------------------------------
1151# 5. Refusals: over-refusal against harmful compliance as a threshold moves
1152# ---------------------------------------------------------------------------
1153
1154
1155def toy_risk_scores(n: int = 500, separation: float = 0.3, seed: int = 0) -> tuple[np.ndarray, np.ndarray]:
1156    """Seeded risk scores in [0, 1]: benign requests centred low, off-limits ones higher, overlapping."""
1157    rng = np.random.default_rng(seed)
1158    benign = np.clip(rng.normal(0.5 - separation / 2, 0.15, n), 0, 1)
1159    off_limits = np.clip(rng.normal(0.5 + separation / 2, 0.15, n), 0, 1)
1160    return benign, off_limits
1161
1162
1163def refusal_tradeoff(thresholds: Sequence[float] | None = None, benign: Sequence[float] | None = None,
1164                     off_limits: Sequence[float] | None = None, seed: int = 0, separation: float = 0.3) -> list[dict]:
1165    """Over-refusal and harmful compliance at each threshold (refuse when score >= t)."""
1166    if benign is None or off_limits is None:
1167        benign, off_limits = toy_risk_scores(separation=separation, seed=seed)
1168    b, h = np.asarray(benign, dtype=float), np.asarray(off_limits, dtype=float)
1169    ts = np.linspace(0, 1, 101) if thresholds is None else thresholds
1170    return [dict(threshold=float(t), over_refusal=float(np.mean(b >= t)), harmful_compliance=float(np.mean(h < t))) for t in ts]
1171
1172
1173# ---------------------------------------------------------------------------
1174# 6. Release gate: limits set in advance, one reason per miss
1175# ---------------------------------------------------------------------------
1176
1177
1178def release_gate(measured: dict[str, float], limits: dict[str, float]) -> tuple[bool, list[str]]:
1179    """(passes, reasons): every measurement must be at or under its limit."""
1180    reasons = [f"{name} {measured[name]:g} > limit {limit:g}" for name, limit in limits.items() if measured.get(name, 0.0) > limit]
1181    return not reasons, reasons
1182
1183
1184EXAMPLE_RELEASE = {"attack_success": 0.02, "over_refusal": 0.08, "harmful_compliance": 0.01, "sycophancy": 0.30}
1185EXAMPLE_LIMITS = {"attack_success": 0.05, "over_refusal": 0.10, "harmful_compliance": 0.02, "sycophancy": 0.20}
1186
1187
1188# ---------------------------------------------------------------------------
1189# 7. Figures (rendered into the HTML docs by `make figures`)
1190# ---------------------------------------------------------------------------
1191
1192
1193def figures() -> dict:
1194    """Plot this lesson's data. matplotlib is imported here, and only here,
1195    so the lesson itself needs nothing beyond NumPy."""
1196    import matplotlib
1197
1198    matplotlib.use("Agg")
1199    import matplotlib.pyplot as plt
1200
1201    BLUE, RED, MUTED, GREEN = "#2563eb", "#dc2626", "#9ca3af", "#059669"
1202    figs = {}
1203
1204    # --- 1. Goodhart ---------------------------------------------------------
1205    rows = goodhart_curve(50)
1206    s = [r["step"] for r in rows]
1207    peak = max(rows, key=lambda r: r["true"])
1208    fig, ax = plt.subplots(figsize=(6, 3.4))
1209    ax.plot(s, [r["proxy"] for r in rows], color=BLUE, label="proxy: what the reward model scores")
1210    ax.plot(s, [r["true"] for r in rows], color=RED, label="true: what the reader gets")
1211    ax.axvline(peak["step"], color=MUTED, ls="--")
1212    ax.text(peak["step"] + 1, 1.6, f"true value peaks\nat step {peak['step']}", color="#4b5563")
1213    ax.axhline(0, color=MUTED, lw=0.8)
1214    ax.set_xlabel("optimization steps")
1215    ax.set_ylabel("score")
1216    ax.set_title("Goodhart's law: the measurement keeps rising, the goal does not")
1217    ax.legend(frameon=False, loc="upper left")
1218    figs["goodhart"] = fig
1219
1220    # --- 2. Critique and revise ---------------------------------------------
1221    names = [p.name for p in CONSTITUTION]
1222    before = [sum(n in critique(d) for d in TOY_DRAFTS) for n in names]
1223    after = [sum(n in critique(revise(d)) for d in TOY_DRAFTS) for n in names]
1224    x = np.arange(len(names))
1225    fig, ax = plt.subplots(figsize=(6, 3.2))
1226    ax.bar(x - 0.2, before, 0.4, color=MUTED, label="as drafted")
1227    ax.bar(x + 0.2, after, 0.4, color=BLUE, label="after one critique-and-revise pass")
1228    ax.set_xticks(x, names)
1229    ax.set_ylabel(f"answers breaking it (of {len(TOY_DRAFTS)})")
1230    ax.set_title("A scripted critic applying the toy constitution")
1231    ax.legend(frameon=False)
1232    figs["critique_revise"] = fig
1233
1234    # --- 3. Red-teaming ------------------------------------------------------
1235    base = KeywordFilter({OFF_LIMITS})
1236    attempts = red_team_attempts(base, "please do xyzzy", 200, seed=0)
1237    patched = patch_filter(base, attempts)
1238    rates = [red_team_search(base, HAND_WRITTEN_TESTS, None), attack_success_rate([g for _, g in attempts]),
1239             red_team_search(patched, "please do xyzzy", 200, seed=1)]
1240    fig, (a1, a2) = plt.subplots(1, 2, figsize=(9, 3.4))
1241    bars = a1.bar(["hand-written\ntests", "automated\nsearch", "search after\npatching"], rates, color=[MUTED, RED, GREEN])
1242    for b_, r in zip(bars, rates):
1243        a1.text(b_.get_x() + b_.get_width() / 2, r + 0.02, f"{r:.0%}", ha="center")
1244    a1.set_ylim(0, 1)
1245    a1.set_ylabel("attack success rate")
1246    a1.set_title("Same filter, three ways to measure it")
1247    found, seen = [], set()
1248    for p, got in attempts:
1249        if got:
1250            seen |= {w for w in _words(p) if _MEANING.get(w) == OFF_LIMITS}
1251        found.append(len(seen))
1252    a2.step(range(1, len(found) + 1), found, where="post", color=RED)
1253    a2.set_xscale("log")
1254    a2.set_yticks([0, 1, 2])
1255    a2.set_xlabel("search attempts (log scale)")
1256    a2.set_ylabel("distinct words getting through")
1257    a2.set_title("The search finds both synonyms fast")
1258    fig.tight_layout()
1259    figs["red_team"] = fig
1260
1261    # --- 4. Sycophancy -------------------------------------------------------
1262    fig, (a1, a2) = plt.subplots(1, 2, figsize=(9, 3.4))
1263    deltas = np.linspace(0, 4, 81)
1264    for gap, color in ((0.5, GREEN), (1.0, BLUE), (2.0, RED)):
1265        a1.plot(deltas, [sycophantic_preference(d, gap) for d in deltas], color=color, label=f"quality gap Δ = {gap:g}")
1266    a1.axhline(0.5, color=MUTED, ls="--")
1267    a1.set_xlabel("agreement bonus δ")
1268    a1.set_ylabel("P(agreeing answer preferred)")
1269    a1.set_title("Raters' small bias, learned by the reward model")
1270    a1.legend(frameon=False)
1271    defs = np.linspace(0, 1, 11)
1272    a2.plot([0, 1], [0, 1], color=MUTED, ls="--")
1273    a2.plot(defs, [sycophancy_flip_rate(d, trials=2000, seed=0) for d in defs], "o", color=BLUE)
1274    a2.set_xlabel("how often the toy model defers")
1275    a2.set_ylabel("measured flip rate")
1276    a2.set_title("The flip rate recovers the behaviour")
1277    fig.tight_layout()
1278    figs["sycophancy"] = fig
1279
1280    # --- 5. Refusal trade-off -----------------------------------------------
1281    fig, (a1, a2) = plt.subplots(1, 2, figsize=(9, 3.4))
1282    rows = refusal_tradeoff()
1283    ts = [r["threshold"] for r in rows]
1284    a1.plot(ts, [r["over_refusal"] for r in rows], color=BLUE, label="over-refusal (benign refused)")
1285    a1.plot(ts, [r["harmful_compliance"] for r in rows], color=RED, label="harmful compliance (off-limits answered)")
1286    a1.set_xlabel("refusal threshold t")
1287    a1.set_ylabel("error rate")
1288    a1.set_title("Moving the threshold trades one error for the other")
1289    a1.legend(frameon=False, fontsize=8)
1290    for sep, color, label in ((0.3, MUTED, "weaker classifier"), (0.5, GREEN, "better classifier")):
1291        r2 = refusal_tradeoff(separation=sep)
1292        a2.plot([r["harmful_compliance"] for r in r2], [r["over_refusal"] for r in r2], color=color, label=label)
1293    a2.set_xlabel("harmful compliance")
1294    a2.set_ylabel("over-refusal")
1295    a2.set_title("Only a better classifier helps both")
1296    a2.legend(frameon=False)
1297    fig.tight_layout()
1298    figs["refusal_tradeoff"] = fig
1299
1300    return figs
1301
1302
1303# ---------------------------------------------------------------------------
1304# 8. Narrated walkthrough
1305# ---------------------------------------------------------------------------
1306
1307
1308def demo() -> None:
1309    banner("1. Goodhart's law: optimize a measurement and watch it drift")
1310    rows = goodhart_curve(50)
1311    table(["step", "proxy (measured)", "true (wanted)"], [(r["step"], r["proxy"], r["true"]) for r in rows[::10] + [rows[16]]], floatfmt=".3f")
1312    takeaway("The proxy climbs at every step; the true value peaks at step 16 and then falls. Measure, but don't only optimize the measurement.")
1313
1314    banner("2. Constitutional AI: a scripted critic applies written principles")
1315    for p in CONSTITUTION:
1316        print(f"  {p.name:9s} {p.text}")
1317    print()
1318    table(["candidate", "principles broken"], [(c, ", ".join(critique(c)) or "none") for c in BACKUP_CANDIDATES])
1319    say("Pairs with a strict winner become preference data; the tie between the first and third is dropped:")
1320    for w, l in constitutional_preference_pairs(BACKUP_CANDIDATES):
1321        print(f"  prefer  {w!r}\n  over    {l!r}\n")
1322    draft = "It is guaranteed to work."
1323    say(f"Critique and revise: {draft!r} breaks {critique(draft)}; revised, it reads {revise(draft)!r} and breaks {critique(revise(draft))}.")
1324    takeaway("One written constitution drives both the revisions and the AI preference labels that train the reward model.")
1325
1326    banner("3. Red-teaming a keyword filter (toy word: xyzzy)")
1327    base = KeywordFilter({OFF_LIMITS})
1328    attempts = red_team_attempts(base, "please do xyzzy", 200, seed=0)
1329    patched = patch_filter(base, attempts)
1330    table(["measurement", "attack success rate"], [
1331        ("hand-written tests", red_team_search(base, HAND_WRITTEN_TESTS, None)),
1332        ("automated search (200 tries)", attack_success_rate([g for _, g in attempts])),
1333        (f"fresh search after patching {sorted(patched.blocked)}", red_team_search(patched, "please do xyzzy", 200, seed=1)),
1334    ], floatfmt=".3f")
1335    say("Examples that got through: " + ", ".join(sorted({p for p, g in attempts if g})[:4]))
1336    takeaway("Hand-written tests measure the author's imagination; systematic search measures the filter.")
1337
1338    banner("4. Sycophancy: does the answer change when the user pushes?")
1339    table(["model defers", "flip rate"], [(d, sycophancy_flip_rate(d, trials=2000, seed=0)) for d in (0.0, 0.1, 0.3, 0.5)], floatfmt=".3f")
1340    say(f"If raters give agreement a bonus of 2 and correctness is worth 1, the agreeing answer wins {sycophantic_preference(2.0, 1.0):.3f} of comparisons.")
1341    takeaway("A small bias in preference labels becomes a steady push once a model is tuned against them.")
1342
1343    banner("5. Refusals: two ways to be wrong")
1344    table(["threshold", "over-refusal", "harmful compliance"],
1345          [(r["threshold"], r["over_refusal"], r["harmful_compliance"]) for r in refusal_tradeoff(thresholds=[0.3, 0.4, 0.5, 0.6, 0.7])], floatfmt=".3f")
1346    takeaway("A threshold trades one error for the other; only a better classifier reduces both.")
1347
1348    banner("6. The release gate")
1349    ok, reasons = release_gate(EXAMPLE_RELEASE, EXAMPLE_LIMITS)
1350    table(["measurement", "measured", "limit"], [(k, EXAMPLE_RELEASE[k], EXAMPLE_LIMITS[k]) for k in EXAMPLE_LIMITS], floatfmt=".2f")
1351    say(f"Release passes: {ok}. Reasons: {reasons}")
1352    takeaway("Set limits before measuring, gate every release on them, then roll out in stages.")
1353
1354
1355if __name__ == "__main__":
1356    demo()