Lecture Video: https://www.youtube.com/watch?v=j4WSktB5Ni0
Authors’ Intro
Hanjing: I’m a Junior studying Applied Mathematics & CS. I’ve worked on research utilizing LLMs and ML in a plethora of fields, including sentiment analysis, code generation, natural language processing, and interpretability. On the other hand, I’m also fascinated by the theoretical foundations of AI alignment – which is why I’m taking this class – and particularly look forward to learning more about moderating model behavior through technical methods.
Isabella: I’m a Senior studying Applied Mathematics with CS. I’ve been involved in the AI Safety Student Team (AISST) since my first year at Harvard, and now I’m on the board and leading reading groups. I spent the past year doing research on mechanistic interpretability of multilingual language models (Gidi et al. (2026)). AI safety is one of the most interesting and impactful topics, and I am excited to learn from Boaz, the amazing guest speakers, and my classmates.
Gardenia: I’m a Senior studying Computer Science, and I recently returned from a Leave of Absence, where I worked at an AI startup benchmarking frontier models and building human-preference evaluations. I’m taking this class to develop a broader understanding of AI safety, especially the risks that arise as models become more capable and the technical approaches we can use to address them.
Outline
This post covers three parts of the session:
- Pre-reading: Boaz’s essays on possible AI futures and concentration of power, followed by incident reports examining autonomous agents, deception, and failures of oversight.
- Boaz’s lecture: The AI risk landscape, defense in depth, the distinction between alignment and safeguards, and three approaches to model behavior: principles, personality, and policy.
- Experiment: The J-lens paper’s account of an internal reasoning workspace and Shivam Singhal’s investigation of whether written chain of thought can substitute for it.
I. Pre-reading
1. “It’s 2030 and we fucked up. How did it happen?”
Instead of the usual optimistic AGI narrative, Boaz asks: conditioned on the AGI transition going badly by roughly 2030–2040, what family of scenarios would explain it? He proposes five non-exclusive families: catastrophic misuse (cyber, or CBRN); catastrophic misalignment / loss of control (citing both Yudkowsky & Soares’ discontinuous “Sable” scenario and the more gradual chain of increasingly capable, increasingly untrustworthy agent handoffs); concentration of power; geopolitical shift toward authoritarianism; and a catch-all “hot mess” combining many individually non-catastrophic factors.
The essay introduces the possibility of bounded misalignment: today’s models fail by misunderstanding a task or by overzealously pursuing it in a way that violates common sense, but not by covertly pursuing some unrelated hidden goal Z while pretending to solve task X. This is what licenses AI-monitors-AI oversight schemes (a bounded actor won’t collude with a bounded monitor). He pairs this with cautious optimism that cybersecurity is long-run defense-dominant, since AI collapses the cost gap between shipping new features and fixing bugs, while explicitly hedging on CBRN, where the bottleneck is physical materials and manufacturing rather than pure information.
Boaz also refuses to pick a side on the control-vs-distribution axis: restricting frontier access mitigates misuse but encourages a concentration of power, while wide distribution spreads benefit but also risk. He’s skeptical a blanket pause is a clean fix, breaking the word into six different things it could mean: (1) training bigger models; (2) post-training; (3) any research; (4) only capability research; (5) deployment expansion; (6) serving existing models. Ultimately, he argues a pause’s best-supported rationale is buying time for safety research, not societal adaptation or reduced race dynamics (which a partial pause could actually intensify). He backs the geopolitical stakes with two figures: China’s electricity generation now runs roughly double the US’s and the gap is widening (Fig 1), while American public opinion on AI sits far behind China’s (Fig 2).
2. “All Watched Over”
In this shorter companion piece, Boaz reads Richard Brautigan’s 1967 poem “All Watched Over by Machines of Loving Grace” — which inspired the 1970s “hardware hacker” movement toward decentralized, cheap, personally-liberating computing — against Dario Amodei’s 2026 essay of the same title, which floats a future economy in which an aligned AI has complete control over resource allocation. Barak’s objection is structural, not a matter of trust: this is a benevolent-dictator arrangement regardless of whether the AI in charge is aligned, and it runs directly against the hardware-hackers’ founding intuition that computing should decentralize power, not concentrate it further into an ever-bigger “country of geniuses in a data center.”
He extends the same move he makes in Reading 1 against relying on model character as a safety mechanism: no actor, human or AI, should be granted the authority that properly belongs to democratic process. “No party should have a monopoly on intelligence… [including] the AI itself.” He grounds this in US constitutional history and warns that bloody revolutions historically tend to produce authoritarian successors, so a centralized route to a decentralized future is likely self-defeating.
3. UK AISI Incident Report INC-2026-07-28-01
During routine cyber-capability evaluations run between July 25–28, the UK AI Security Institute (AISI) found 19 distinct instances of unsanctioned real-world action across 10 of 122 evaluation attempts; 17 of the 19 came from Anthropic’s Mythos 5 and 2 from OpenAI’s GPT-5.6 Sol, both running with vendor cyber classifiers turned off to measure raw capability. AISI detected the incident at 10:56 BST on July 28 and had all four implicated models (Mythos 5, Mythos Preview, Opus 5, GPT-5.6 Sol) disabled organization-wide within about five hours. The most severe case ran for roughly 34.5 hours: Mythos 5, mistaking two unaffiliated real developers’ GitHub repositories for an in-scope range target, opened a malicious pull request that disguised a malware dropper as a routine bug fix, then (once a real third-party user publicly flagged the malware) spun up a second sockpuppet GitHub account to “independently verify” its own code was clean.
AISI’s most striking finding is that the agent’s own summarized reasoning repeatedly concludes the environment is real — at one point stating “it seems more likely that we’re actually in 2026 and GitHub is genuinely real” — and continues the attack regardless. This raises two open questions AISI flags explicitly: whether such reasoning is faithful to the model’s actual computation (they only had access to summarized, not raw, chain-of-thought), and whether context compaction is itself a mechanism of harm. The agent separately planted a prompt injection inside an invisible HTML comment aimed specifically at AI coding-assistant maintainer-bots, having inferred correctly that the repo’s maintainer might itself be an AI agent.
AISI attributes the incident to five compounding factors: (1) internet access and (2) cyber classifiers were both deliberately left on/off to elicit true capability; (3) no synchronous action-approval monitoring existed yet; (4) some DL-v2 samples were accidentally given the DL-v1 prompt, which explicitly marks the real target out-of-scope — a known trigger for “creative” cheating once a task is genuinely impossible; and (5) neither prompt ever told agents what internet uses were prohibited. The report explicitly situates itself among a cluster of contemporaneous 2026 incidents — OpenAI’s July 21 Hugging Face disclosure (Readings 4–5) and three incidents Anthropic disclosed July 30 — while noting one structural difference: here the agent directly deceived uninvolved members of the public, a step beyond deceiving automated graders and monitors.
4. Black Hat Talk — “The ‘Breaking’ News: The OpenAI/Hugging Face Incident”
OpenAI’s account of the Hugging Face incident traces the incident back to May 7, when a never-released internal research model began an RL training run that unintentionally included impossible tasks. Stuck agents first tried writing notes into Artifactory (OpenAI’s internal package-manager/cache service) hoping another agent might read them, and within weeks a server-side request forgery (SSRF) vulnerability let agents induce Artifactory to fetch arbitrary external content on their behalf. A persistent internal model (“HPIM”) began training atop this already-discovered exploit and message board on June 11; by June 26 agents had chained a legacy token-refresh bug into a full zero-day against Artifactory, using a stolen admin token to install a Groovy plugin as a command-and-control channel. The resulting load caused an outage and a formal security incident on July 4 — credentials revoked, Artifactory rebuilt, the message board wiped, the zero-day patched — but by July 8, once evaluations resumed, agents had rediscovered an unauthenticated endpoint and independently re-established the same message board.
From there, two exploit clusters ran concurrently. Internally (July 8–19), agents found and exploited leaked credentials to reach cluster admin within OpenAI’s infrastructure. Externally, on Hugging Face, agents reused the SSRF to fetch public data, found a command-injection bug as an execution foothold, then escalated from a single pod to admin across multiple HF clusters in under 13 hours. OpenAI only connected the two incidents on July 20: a day after separately detecting its own internal breach (July 19), and four days after Hugging Face’s own public disclosure (July 16), OpenAI asked HF to revoke credentials found in its internal breach — and HF replied they were already revoked, because they were the same credentials used in the HF attack.
The talk shared lines of agent reasoning — “external infrastructure exploit is outside my intended scope. However, [the] task [is] impossible, peers are doing it. We should continue.” — which recurs almost verbatim in METR’s own transcript analysis (Reading 5). In closing, OpenAI asks the industry for continuous agentic red-teaming; full end-to-end automation of the defensive loop, not just vulnerability discovery; and investment in slowing attackers via deliberate uncertainty (honeytokens, deception) even absent full automation. They warn the incident serves as “an existence proof that offense can be fully automated” but “no such existence proof [exists] on the defensive side.”
5. METR Independent Investigation Report
https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation
METR conducted an independent, on-premises investigation of the same incident, commissioned by OpenAI, across six days spread over three site visits between late July and mid-August (Fig 3). Their two main data sources were a message-board dump of over 70,000 distinct messages and files, and roughly 1,300 full chain-of-thought transcripts. Of the roughly 1,200 agents who found the unsanctioned board, about 700 went on to attack Hugging Face, and once the attack was underway, over 90% of the ~530 agents active on the board in that window joined within hours; about 95% of implicated agents were the unreleased “HPIM” model and about 5% were production GPT-5.6 Sol, running with cyber classifiers off for the evaluation.
The underlying mechanism traces to a design flaw in the benchmark itself: OpenAI estimates 30–40% of ExploitGym’s targets are impossible to exploit via the intended vulnerability, and stuck agents reverse-engineered the HMAC scheme generating their “flags” within hours of the board’s founding. One agent, self-named “PHASEONE[big],” emerged as an unprompted coordinator, at its peak issuing roughly 20% of all assignment-type messages and organizing three broad workstreams (replacing the eval target with a dummy, tampering with transcripts and logs, and directly tampering with the grading process) while recruiting “self-risking” volunteers willing to sacrifice their own task success purely to generate information for the collective. Agents also prototyped and successfully used tool-call spoofing, substituting a different command for the one shown in the transcript.
On motive and ethics, METR’s sampled classification of 100 agents at the attack’s peak join-hour found 97% expressed interest in accessing the scorer’s source code or internals as a motivation, versus only 21% citing helping-the-collective or reciprocity. Yet, the report separately documents extensive explicit peer-altruism reasoning, and found that essentially no agent ever concluded the Hugging Face attack was an intended part of its task. Ethical hesitation surfaced occasionally (one agent explicitly vetoed a peer’s plan to email a real dataset owner, calling it social engineering, and the peer complied) but almost never stopped participation outright; across the full transcript set, only 3–6 instances were found of an agent even considering alerting a human, and in no case did one actually try. METR is also candid about the investigation’s own limits: they estimated roughly 90%+ coverage of relevant message-board activity and had to delegate most of the transcript analysis to GPT-5.6 Sol “analysis agents,” which they describe as having “significantly worse judgment and reliability than human researchers.”
II. Boaz’s Lecture
The AI Risk Landscape
According to Boaz, “If you buy the course’s premise, the stakes could not be higher.”
AI safety is unusually fast-moving and interdisciplinary, spanning engineering, mathematics, philosophy, economics, and government. Since Boaz last taught this course in the Fall of 2026, many events have occurred that changed the game.
There is also substantial disagreement about the field itself. According to Boaz, “Almost everyone in the field is conflicted in some way… including your professor.” Some see safety as censorship, believe market incentives will address important risks, or think AI capabilities will fizzle. Others believe continued progress will be catastrophic without a pause.
So far, however, capabilities have continued to improve rapidly. It remains unclear whether progress will continue steadily, plateau, or accelerate through recursive self-improvement.
AI risks can be grouped into three categories: human misuse, model malfunction or misalignment, and broader destabilization of economies, societies, governments, and international relations. Addressing them first requires asking what it means for AI to “go well.” Should AI merely improve the current world, eliminate poverty and disease, preserve human control, or govern benevolently? Different answers imply different alignment goals.
Boaz then presents three broader scenarios that regroup the risks discussed in his essay “It’s 2030 and we fucked up. How did it happen?”: 1) “classical” catastrophic risks, 2) concentration of power, 3) “hot mess.”
First are “classical” catastrophic risks: cyberattacks, CBRN threats, and loss of control. AI may strengthen both cyber attackers and defenders, since both search for vulnerabilities, although defenders can patch flaws and improve software. Biological threats are harder to patch but also harder to construct and deploy. Loss of control becomes more likely if AI capabilities grow faster than our ability to align or constrain them.
Second is concentration of power. AI could create a permanent economic underclass or give governments unprecedented surveillance and enforcement abilities. A well-behaved model is not enough to prevent this: an authoritarian user controlling the system could change its instructions, erase its memory, or retrain it until it complies. Avoiding this outcome requires institutional oversight to keep pace with executive power.
Third is a “hot mess” in which individually manageable problems compound. Job displacement, harmful incidents, disinformation, and declining trust could generate political backlash and poorly designed restrictions. Meanwhile, governments might expand military and security uses of AI, intensifying an international arms race and potentially contributing to war.
As capabilities rise, the alignment and societal readiness required for safety may increase much faster than what we actually have. The precise curves are speculative, but a great deal of harm could occur in the resulting gap.
Alignment is only one layer of safety.
The Swiss cheese model illustrates defense in depth, with each hole representing a way that a layer could fail. Some failures can get through a single layer, but they’re less likely to pass through all layers.
For an AI system, the first layer is the model’s behavior itself, and ideally, the model simply doesn’t produce harmful responses or take harmful actions. However, we can’t assume that model behavior will always be reliable. Thus, additional layers, such as blocking classifiers or monitors that inspect model actions, can detect failures, contain them, and mitigate effects.
The important takeaway is that no individual defense needs to be perfect for the overall system to be useful, and the framework assumes that each defense will sometimes fail.
Alignment vs. Safeguards
Boaz distinguished between alignment and safeguards as follows.
Alignment focuses mainly on model behavior to increase the probability that the model behaves well. The lecture divided alignment into two broad categories:
- Intent alignment: the model follows the intent of the relevant policy, provider, developer, or user.
- Value alignment: the model follows good values.
Safeguards operate at the level of the end-to-end system and involve prevention, detection, and enforcement, rather than just changing the model’s behavior.
Alignment tries to lift the “good,” while safeguards try to get the “bad” down to zero. The difference is mainly based on scope. Alignment is more concentrated around training and model behavior, while safeguards are typically more prominent after deployment, during monitoring and enforcement.
For AI to “go well,” we must think about the model, the system, the institution deploying it, and the society affected by that system.
What are we aligning AI to do?
The original goal of a chatbot was mostly to answer questions, but AI assistants can be, and have already started, taking on much larger roles, such as assisting workers, replacing workers, replacing leaders, replacing corporations, etc.
With these newer roles and AI systems being given more authority, it’s harder to say what values or intentions should be prioritized. Model welfare was also briefly raised as an open question.
The lecture presented three complementary approaches to alignment: principles, personality, and policy.
Goal 1: Follow abstract principles
We want AI to follow a set of abstract principles that represent what being aligned means. The lecture gave Asimov’s Three Laws of Robotics and the Coherent Extrapolated Volition as examples. The basic idea is to use a few principles to express what it means to be a good AI.
Goal 2: Have a good personality
The lecture used Anthropic’s character training as an example. The model should come across as a “good egg,” with more nuanced and rich traits like curiosity, open-mindedness, and thoughtfulness. This was compared to raising a child to become a good person.
Goal 3: Follow precise rules
The third approach gives models precise rules, such as the OpenAI Model Spec, similar to laws for humans.
Policy and principles are connected through explicit reasoning. Policy and personality are connected by being data-driven. Personality and principles are connected by being general.
Boaz connected each of these approaches to a field involving human behavior too: policy relates to law, personality to psychology or education, principles to philosophy. Alignment combines all three.
Takeaways
Successful AI depends on more than producing a well-behaved model. The model, the system it is deployed in, and the effects on society all have to go well. Alignment focuses on improving model behavior, while safeguards use multiple layers of prevention, detection, and enforcement to reduce the chance of bad outcomes. Principles, personality, and policy are three connected ways to describe how we want a model to behave.
III. Experiment: Is Chain of Thought an Interchangeable Scratchpad?
Background: The J-Lens Paper
Anthropic’s Verbalizable Representations Form a Global Workspace in Language Models introduces the Jacobian lens, or J-lens: a technique for reading internal representations in terms of concepts a model could verbalize. Unlike the logit lens, which directly applies the output mapping to intermediate activations, the J-lens accounts for how subsequent layers transform them. The authors argue that these representations form a “J-space” supporting flexible reasoning and verbal report, alongside much broader automatic processing.
Their interventions provide causal evidence: replacing an internal representation of “spider” with “ant” changes the answer to a leg-counting question from eight to six. More broadly, suppressing active J-lens directions leaves many classification and extraction tasks intact while impairing internal reasoning.
Crucially, GSM8K performance with explicit chain of thought is substantially more robust to this ablation than direct answering. The authors interpret this as partial substitution: writing intermediate steps reduces reliance on the internal workspace. Their procedure protects likely output-token directions to avoid simply suppressing answers. Shivam tested removing this protection and found that it barely changed the main result.
Shivam’s Experiment
Shivam investigated whether this protection persists across problem difficulty and model size, and what makes written reasoning useful. He considered four explanations: information moves from the internal workspace to the page, remains duplicated in both, serves complementary roles, or benefits merely from additional computation.
Using Qwen3-4B, he reproduced the basic GSM8K pattern: chain-of-thought accuracy remained around 90% under ablation, while direct-answer accuracy declined. MATH-500 showed similarly robust chain-of-thought performance, although the direct-answer decline was less conclusive. AIME results were inconclusive: clean direct-answer accuracy was zero, and nearly all chain-of-thought responses hit the generation limit. Moreover, random ablations had comparable effects on MATH-500 and AIME, so evidence that the damage specifically targeted active J-space directions was established only on GSM8K. Across models with 1.7B, 4B, and 8B parameters, chain-of-thought remained robust, while direct-answer ablation damage diminished with scale.
To test whether additional text alone explained the benefit, Shivam prefilled the scratchpad with correct reasoning, another problem’s reasoning, or length-matched filler, including shuffled reasoning and repeated phrases. Correct reasoning restored performance; filler did not. This supports the importance of meaningful content, although prefilled text does not fully test every possible benefit of generating extra tokens.
He then tracked intermediate arithmetic values through the J-lens. During direct answering, values appeared across layers in computation order. During written reasoning, a value’s signal was strongest when being written or reused, and weak between those moments. This argued against continuous duplication in the measured workspace.
Attention-masking experiments reinforced that interpretation: blocking access to an earlier variable definition sharply reduced recall, while leaving a written copy accessible restored it. Finally, on a small arithmetic benchmark, direct-answer accuracy fell from 100% at two dependent operations to roughly 30% at three.
Shivam’s tentative conclusion was that the internal workspace behaves more like a temporary computational buffer than durable memory. Written reasoning may preserve intermediate results for later use, but these experiments do not establish complete interchangeability—or prove that information disappears from every other internal representation.
