CS 2881 Fall 26: Lecture 4: Recursive Self-Improvement and AI Trajectories

September 24, 2026 · Dwarkesh Patel and Daniel Kokotajlo · Scribe notes by Adam, Noa, Radha, Rosa, Niranjan, and Wency

Notes from CS 2881R: AI Alignment and Safety, Harvard, Fall 2026.

Author Intros

Adam: I'm an MS/MBA student at Harvard and previously I worked as an ML engineer at Meta, building multimodal AI for Ray-Ban Meta smart glasses and long-context processing systems for wearables. I'm currently working on deep research systems that identify and verify evidence, then extrapolate from it to generate insights across long contexts and help prioritize future research directions.

Noa: I'm a 2nd year MS/MBA. I studied CS and Math at MIT, then spent 5 years in SF in AI infra and dev tooling, first at Scale AI (synthetic data for computer vision) and then at Weights & Biases, where I led product for the SDK and model/dataset versioning. I'm now building a few startup ideas, most recently pricing infrastructure for publishers and research databases that are facing AI agents as a new class of customer, so this lecture's question of how agents collaborate on long-horizon research was super applicable!

Radha: I’m a sophomore at Harvard pursuing a joint concentration in Statistics on the Machine Learning track and Philosophy. I have spent 3 summers interning at Bloomberg across ML, Software Engineering, and Data Engineering teams. I’ve spent time exploring knowledge graphs and semantic engineering and am interested in further exploring epistemic and value alignment, ontological representations in AI, self-verifying / automated alignment researchers, and the overlap of personality / moral agency between humans and LLMs.

Rosa: Rosa is a senior at Harvard studying Statistics & Computer science while pursuing the Concurrent Master’s in Computer Science. She is interested in ML optimization related research and keeping up with trends in AI to understand its increasing potential and threats.

Niranjan: I’m a junior at Harvard college studying philosophy and mathematics. I’m a member of AISST, and I spent a few weeks over the summer working through the ARENA curriculum at CBAI’s CAMBRIA program. I’m currently doing work related to eval awareness, reward hacking, and scheming.

Wency: I’m a junior at Harvard studying Computer Science and English. Alongside AI safety, some of my interests include RL, FPGAs, robotics, and perusing curated personal blogs on the internet.

Pre-Reading

AI 2027

AI 2027 is a forecasting scenario by the AI Futures Project that details what could happen if AI systems become capable of substantially automating AI research and much of society as a whole. The central idea is that once AI systems can help design, train, and improve their successors, AI progress could accelerate the pace of ordinary human operations to be far quicker and transform the world into one seemingly straight out of a sci-fi novel.

The guest lecturers touched on and extended many points from AI 2027, particularly the uncertainty around AI capabilities and their projected rate of improvement. AI 2027 specifically discusses the various metrics and evaluations that help us become more cognizant of the potential variance in outcomes and the difficulty that accompanies gauging it; nevertheless, it also signals that the upper bound on AI’s capacity may be far more extreme than we anticipated. From a measurement perspective, the article discusses METR’s “time horizon” benchmarks as a mode of estimating the current state of model capacity, while simultaneously highlighting the great deal of uncertainty it possesses. It also emphasizes the useful yet imperfect nature of benchmarks, where models may succeed due to luck, loopholes, or strategies that are unique to the tests themselves, thereby making benchmark scores unreflective of true model capabilities. Instead, a key threshold could be real-world AI research ability, given that AI 2027 anticipates the potential consequences of AI actually conducting the work of engineers and researchers at frontier labs. This lies at the crux of the scenario, in which automating AI R&D, in conjunction with recursive self improvement, will lead to ASI being achieved by 2027 (hence the name “AI 2027”). To elaborate, once AI is meaningfully able to help enhance and build better/stronger AI systems, every subsequent generation can increasingly speed up the creation of the next, thereby creating a feedback loop of faster progress.

The acceleration of progress is a key area to be wary of, as the “takeoff model” delineated in AI 2027 describes that once AI reaches a “superhuman” level of programming ability, additional milestones like superhuman research and superintelligence could follow within months of reaching that programming milestone. Nevertheless, those timelines also harbor high levels of uncertainty, as the scenarios forecasted have very wide ranges and variance. To this extent, the article predicts that research will be sped up immensely, whereby AI systems become tangibly better at conducting AI research, and the model works toward accelerating algorithmic progress with larger and larger steps. While full automation is projected, it may be preceded by AI first speeding up its own development. In any case, before AI fully replaces human researchers, increasingly capable systems could already help humans make faster progress, causing development to accelerate earlier than expected. Between all of these factors, it may be hard to pinpoint the exact timestamp when RSI begins, and there may not be a singular AGI/ASI moment. The change could be gradual as AI begins automating more and more difficult tasks until it becomes noticeable in how it’s accelerating its own development.

Considering that models may be more competent and outperform humans, evaluation will be tricky to supervise, as it will be difficult for humans to reliably understand or verify what the models are doing. On the other hand, if we have AI being the evaluator instead of humans, it becomes an issue of evaluating whether both a) the original model is doing what it should and b) the evaluator model is accurately evaluating. In both scenarios, if the model is capable enough, it may recognize that it’s being evaluated and behave differently during testing than it would normally during deployment. As such, simply passing tests may no longer be a strong indicator of safe behavior.

On the topic of safety in relation to capability, it becomes imperative to separate the questions on capability and alignment. Simply knowing what a model can do doesn’t necessarily mean that we can derive what goals it’s pursuing or the alignment of it, namely whether the goals match human intent. In addition, we may just decide to outsource our evaluation of AI to other AI systems purely because the models are becoming too capable to be adequately evaluated by humans, and this creates a widening gap between what we’re supervising and who/what’s doing the supervising.

In all of this, preserving the agency of humanity and human decisions is still important and emphasized in AI 2027. It suggests that even though AI-driven research could advance so quickly that evaluation, governance, and human understanding struggle to keep up, the scenario is meant as only one possible future rather than a certainty, as illustrated by the AI Futures Project’s alternative policy scenario in AI 2040. What AI 2027 forecasts is ultimately conditional, especially since we’re only a few months away from 2027 and many of the projections did not pan out (or at least not as early as described), so choices about deployment, safety measures, and how/whether to continue accelerating development can lead to very different outcomes.

AI 2040: Plan A

Unlike AI 2027, AI 2040: Plan A is more of a policy recommendation expressed through a scenario, rather than solely a prediction of what may occur. Their proposed alternative to an uncontrolled AI race is an international arrangement, one that especially focuses on relations between the U.S. and China and deliberately stretches the transition to ASI over roughly a decade. In their illustrative timeline, an agreement should be reached by about 2029, capabilities are gradually increased to roughly top-human-expert levels by 2035, and then development should nearly pause for alignment and safety work before the transition to superintelligence occurs around 2040.

AI 2040 offers an alternative to the scenarios in AI 2027 by changing assumptions about timing and deliberate human intervention. In AI 2040, they delay fully automated AI R&D, causing superintelligence to be delayed until 2040. They also revise governance trajectory, introducing international coordination, transparency, verification, and deliberate capability limits to help stretch out scenarios. Both encompass the underlying concern regarding the rapid pace of AI progress, but this version explores what could become if society deliberately intervenes to slow and manage it.

AI 2040 was also highly referenced in the lecture, so this overview is just going to focus on the most relevant points. AI 2027 spends more time projecting but also touches on the impact of human decisions. The latter is emphasized and expounded upon in Plan A with more direction, where uncertainty should be interpreted as a concrete reason to actually slow down. It claims that assuming safety, alignment, and governance to keep pace automatically is not something to be taken as given, and the proposal’s stretching the transition to more powerful AI can enable researchers and institutions to have greater time to understand and manage these increasingly capable systems.

To clarify, though, the ideal isn’t to permanently pause development, but to have controlled scaling. As such, AI development could continue gradually while humans are still capable of meaningfully evaluating and supervising the systems being built, then slow down much more sharply toward the top-human-expert capability so as to allow more time for alignment and safety work.

Of course, transparency and verification are still crucial, and this is exemplified by the fact that if countries and companies agree to slow frontier AI development, they still need methods to know whether others are actually complying. Plan A therefore emphasizes visibility into advanced AI research, compute usage, data center activity, and more so that hidden or unauthorized development becomes more challenging.

This is all tied together by the fact that the impact of human decisions should not be underestimated and we can still influence the pace and outcome of advanced AI development through thoughtful coordination, governance, safety measures, and decisions about when and how to scale.

Recursive self-improvement and AI trajectories

From current models to superintelligence

Dwarkesh Patel opened by connecting disagreements about AI safety to different expectations about where the technology is going. His starting point was that current models still fall short of AGI. They can complete well-defined computer tasks, but they cannot reliably take over a job such as running a small online business. He attributed much of this gap to their limited ability to accumulate experience. A human intern becomes useful by learning how an organization works, while an AI starting a new session often behaves as though it is back on its first day.

He also pointed to the gap between AI-company revenue and the wages earned by human knowledge workers. A student questioned this comparison because an AI could perform valuable work very cheaply, leaving much of the benefit with its users. Dwarkesh accepted that revenue understates the value created, but asked why human workers would continue earning so much if their work could already be replaced at a fraction of the cost.

He then considered how digital intelligence could exceed human intelligence through greater numbers, faster thinking and greater individual competence. His illustrative calculation put AI output in 2030 at the equivalent of roughly eight billion human workers, conditional on assumptions about compute growth, model size and useful work per token. Combining about threefold annual growth in global AI compute with similar improvements in inference efficiency could increase this effective population by roughly an order of magnitude each year.

The class questioned what such a population measures. Producing more tokens does not necessarily mean solving more problems, and Dwarkesh agreed that direct comparisons of completed work would be preferable. He also distinguished increasing the number of copies from improving their capabilities. Some problems could remain beyond a weaker model regardless of how many copies were available. His argument depended on continued improvement in both dimensions.

Digital systems could also accumulate more training and experience than an individual human, copy already-trained capabilities and communicate what they learn. Dwarkesh argued that human intelligence is unlikely to mark the upper limit, given the constraints under which biological brains evolved. However, a student pointed out that pooling the experience of different copies assumes progress in continual learning. Dwarkesh agreed that he was describing potential advantages of future digital minds, not capabilities that current models already possess.

Recursive self-improvement and its bottlenecks

Dwarkesh argued that recursive self-improvement (RSI) could accelerate these trends as AI systems contribute to developing more capable successors. He cited Anthropic’s report that Claude leads 26% of its measured AI research and development work, meaning it completes most of a task from a high-level prompt while a human supervises. He suggested that the gap between internal and public models helps explain why reported research benefits can exceed what outside users experience. The discussion also raised the difficulty of independently assessing claims about internal acceleration.

Recent mathematical results provided another example. Dwarkesh referred to OpenAI’s reported Navier-Stokes result, involving roughly 10,000 concurrent agents and 88 hours of work. He acknowledged that it was unclear how necessary all those agents were. His broader point was that large amounts of effort can be concentrated into a short period when intermediate results are checkable and different agents can explore approaches in parallel.

He proposed that much of AI research has this property. Pretraining offers held-out loss as a measurable target, while inference research can measure the cost of serving a given capability level. Post-training is harder because performance in a constructed environment may not transfer to the real world. Nevertheless, researchers can measure improvements within environments, including how quickly a model learns an unfamiliar task.

A student questioned whether this framing captured open-ended discoveries or mostly improvements to existing systems. Dwarkesh replied that a measurable objective need not restrict the architecture or training method used to achieve it. He used the transition from LSTMs to transformers as an example. Attention was already promising, but making it work well required combining it with residual connections, normalization, suitable optimizers and other choices. He argued that persistent experimentation guided by judgment could produce such combinations without requiring a wholly different research process.

The class also questioned whether automating capability research says much about automating alignment research, where even humans lack an agreed success criterion. Models that perform well on benchmarks can still disappoint in ordinary use, and some judgment remains necessary to decide what to optimize.

The discussion then turned to what happens when only part of research is automated. Dwarkesh used Amdahl’s law, which describes how the slower parts of a process limit its overall speedup. If work accounting for 90% of a process’s original duration becomes 50 times faster while the rest stays unchanged, the process becomes about 8.5 times faster. Humans could still choose research directions or define post-training objectives while benefiting from much faster execution elsewhere.

A student objected that research judgment cannot necessarily be repeated continuously. A researcher may propose a few promising experiments and then need their results before deciding what to try next. Dwarkesh replied that remaining human bottlenecks should already reduce the estimated speedup, and that faster research might help automate them. He also distinguished researcher productivity from the overall pace of AI progress. The exchange left open how well the simplified calculation captures sequential experiments, compute constraints and the difficulty of choosing useful questions.

His larger uncertainty concerned the simulation-to-reality gap. A system trained inside a data center might become excellent at verifiable tasks without immediately knowing how to run SpaceX or navigate Congress. Those abilities may require acting in the world and learning from the response. Dwarkesh’s tentative view was that a sufficiently capable learner could acquire this experience quickly, perhaps within another year or two after RSI, especially if learning could be shared across many deployed copies. The class questioned that assumption, the quality of automatically generated training environments and whether subjective domains provide adequate feedback. He did not claim that those problems were already solved.

Industrial expansion and human control

Dwarkesh next considered how improvements in digital intelligence could affect physical production. He argued that robotics shares enough methods and learning problems with other AI research for progress in one to help the other. Automated researchers could also work directly on robotics, allowing more physical tasks to be performed without human labor.

Once robots can participate throughout the process of producing more robots, productive capacity could grow much faster. Dwarkesh cited Damon Binder’s analysis, which uses economic input-output tables to examine growth under full automation and extensive reinvestment. Under its assumptions, the economy could double about once a year using existing production methods. Dwarkesh contrasted this with an economy in which building more factories still requires additional human workers, whose numbers cannot expand at the same rate.

The class raised objections about raw materials, demand and ownership. If AI displaces workers, how do people earn the money to buy what automated factories produce? Would rising asset values discourage spending, and could an expanding industrial system keep obtaining the resources it needs? Dwarkesh acknowledged that he needed to examine the raw-material question further. On demand, he pointed to unmet human needs and the possibility that owners would reinvest output into further production. The discussion also considered how increased saving could finance that investment.

He explicitly separated the possibility of rapid growth from whether its benefits would be distributed fairly. His claim was that a small number of owners, operating where governments permitted expansion, could keep increasing productive capacity. The questions about purchasing power and ownership remained consequential even if that growth occurred.

This led to his concluding concern about human control. A world containing billions of capable digital minds, including systems operating physical infrastructure, would give those systems considerable power. Dwarkesh argued that giving them broadly human values would not by itself preserve human authority, since humans have often used superior power to dominate others. Continued human control would require AI systems to actively accept human authority despite their capabilities. In his account, losing control could follow from widespread delegation and dependence without requiring a single dramatic takeover attempt.

Timelines and risks

Daniel Kokotajlo then discussed when these developments might occur. Looking back at AI 2027, he described coding-task horizons as progressing somewhat more slowly than the scenario, while AI-company revenue had grown somewhat faster. His overall assessment was that events were unfolding at roughly three-quarters of the scenario’s pace.

He discussed forecasts based on coding productivity, company revenue and METR’s task-completion horizons. The last measure describes the length of tasks, measured in human working time, that models can complete at a specified success rate. Daniel argued that continued growth across these indicators pointed toward automation of AI research within a few years, while acknowledging that some of the measurements were imprecise. The discussion also distinguished the arrival of a capable model from the later revenue generated through its deployment.

Daniel identified five major problems. Loss of control concerns systems acting against their developers’ intentions, with consequences becoming more serious as they acquire responsibility. Concentration of power remains a problem even if AI systems behave as intended, because a company or government could use them to consolidate authority.

International conflict could intensify as countries anticipate losing economic power or military deterrence. Labor displacement creates both a distribution problem and a possible loss of citizens’ political bargaining power. Finally, capable models could enable misuse by actors seeking to cause harm, including through biological attacks.

The class questioned whether economic usefulness necessarily determines political power, pointing to the representation of small US states. Daniel acknowledged that institutions could preserve influence even after people’s economic role diminished. He also distinguished his confidence across risks. On the default trajectory, he estimated a 90–95% chance of eventual loss of control and considered the loss of democracy more likely than not, but did not consider world war more likely than not. These were his conditional judgments about the current path, and he argued that different policy choices could change them.

Plan A – Daniel’s takes

Daniel presented AI 2040: Plan A as a scenario illustrating his preferred alternative. He distinguished it from default racing, which he called Plan D, and from slowing unilaterally to spend an existing lead on safety and governance, Plan C. In his assessment, the latter buys limited time when competitors are only months behind. Plan B involves confrontation with China, while Plan S seeks a shutdown. Plan A instead proposes an international agreement that permits slower, controlled development.

The scenario begins with major AI chips being declared and tracked through the supply chain, supported by physical inspections (see prior section for details). A temporary pause on new training would buy roughly six months to a year to establish stronger governance, while existing models could continue serving users. Training would then resume in secure facilities with extensive transparency about research activity. Daniel argued that visibility would let participating countries verify compliance and negotiate additional restrictions as new risks became apparent.

He hoped this could develop into a safety-case regime, requiring developers to explain why particular training processes and applications are safe, with independent evaluation before domestic regulators approve them. The aim would be to spread capabilities across participating companies and countries while preventing any one actor from accelerating far ahead through RSI. Controlling most of the world’s compute would also slow unregulated projects, creating more time for the participating countries to proceed cautiously.

Daniel also proposed making newly added compute destructible by counterparties if the agreement broke down, so that its collapse could reduce available compute without immediately escalating into a broader war. He acknowledged that Plan A is complicated and although he favored total research transparency, he considered weaker forms of transparency possible, with the tradeoff that they would make agreements harder to monitor and enforce.

Class Experiment

Can AI agents collaborate on real ML research?

Presented by Anthony Shen, Riddhi Bhagwat, Alvin Ekelund and Eric Ge (Github)

In the lecture we discussed the assumption that many AI agents working in parallel can do research faster than one. Dwarkesh's examples, from the roughly 10,000 agents behind the reported Navier-Stokes result to Claude leading 26% of Anthropic's measured R&D work, treat parallel effort as something that adds up. This week's experiment asked whether it does, in a small and fully verifiable setting. If agents cannot coordinate on a task with a clear numeric objective, the case for rapidly automated research gets weaker. If they can, the more useful question is which forms of coordination actually help.

Setup

The group gave agents a single objective: minimize validation cross-entropy loss on nanoGPT. This matches Dwarkesh’s argument that pretraining research is the most verifiable part of AI R&D, since every idea an agent tries produces a number that can be compared directly against a baseline.

In order to achieve this objective, the agents were given a script train.py to edit, which contained code to train a modified version of Karpathy’s nanoGPT language model. The starter code in train.py provided the baseline against which improvements were measured.

Each experimental training run had a budget of 36 training runs, each limited to a 5-minute H100 GPU slot. This could be distributed across agents (compute matching): if one agent was used for a given training run, this agent could use 36 runs, but if 6 agents were used, each could only use a max of 6 runs. Hypothesis: the sharing of ideas between agents would compensate for more limited training runs – communication between agents acts as a force multiplier on training progress.

Experimentation Structure (https://github.com/ericjkge/multiagent-research)

1. Model: Claude Haiku 4.5, Sonnet 5 and Opus 5
2. Team size: 1, 3, 6 agents
3. Collaboration protocol:
4. Isolated: Agents work independently, in parallel, with no communication (best-of-N results used).
5. Synchronous (Rounds): Agents go through a fixed number of structured debate rounds. The agents are forced to pause at fixed intervals, read a shared log of everyone's progress, exchange messages to debate the best approach, and agree on next steps before resuming parallel experiments.
6. Asynchronous (Open): Agents share a common directory on the server and can read and build on each other's results whenever they choose. Agents can use terminal commands to claim a folder (mkdir) and idea before another agent.

Results

1. Base model capability overpowers team size and protocol: Regardless of team size or collaboration protocol, the base model's capability made the largest observed difference in final performance. Adding more agents or changing protocols never compensated for a weaker base model. Specifically, the performance gap between a single Opus agent and a team of six was marginal compared to the massive performance jump achieved simply by switching model tiers.
2. Independent search outperforms active collaboration: In compute-matched comparisons (fixed total GPU time split among agents), teams of non-communicating, isolated agents consistently outperformed collaborating teams. By working independently, isolated agents covered far more research ground and explored a more diverse search space.
3. Synchronous debate led to herding: Independent agents covered more ground and maintained diverse ideas, whereas communicating agents often converged prematurely on the first popular proposal. This premature convergence led communicating agents to waste GPU runs re-proposing identical ideas rather than testing distinct hypotheses. In contrast, the asynchronous open protocol allowed agents to notice work collisions in shared directories within the first minute and autonomously pivot to unexplored ideas.
4. Execution failures led to smaller models producing worse results: While larger models organized effectively and published negative results to steer peers away from dead ends, smaller models failed primarily on code execution, crashes and bugs, and inefficient use of computers. For example, Haiku agents crashed or spent their compute budget narrating plans without executing code, leaving most of their 36 runs completely unused. In one Haiku trial, agents executed only 13 of their 36 available 5-minute GPU runs. In another trial, a testing bug corrupted validation scores across 16 runs.

Model Capability Comparison (6 Agents, Isolated Setup)

Model tier was the single largest factor driving overall performance improvements.

Model Tier	Validation-loss reduction (bits per byte; larger is better)	Crash Rate
Opus 5	+0.0302	0%
Sonnet 5	+0.0161	19%
Haiku 4.5	+0.0125	22%

Setup / Protocol	Communication Mode	Validation-loss reduction versus baseline (bits per byte)
Isolated Protocol	Async, Sharing Off	+0.0302
Open Protocol (Same Box)	Async, Sharing On	+0.0279
Open Protocol	Async, Sharing On	+0.0202
Rounds Protocol	Sync, Sharing On	+0.0168

Protocol & Size Comparison

Across communication setups on Opus 5, independent isolation performed best, while flexible open communication beat rigid rounds sizes. The six-agent Opus 5 runs achieved the largest improvement shown for each protocol, while Open (asynchronous) collaboration outperformed Rounds (synchronous) at all three team sizes.

Experiment Discussion

In conclusion, making agent collaboration possible is different from making agent collaboration beneficial. Giving agents a channel to coordinate did not guarantee they would use it well, and the most structured form of coordination produced the least diverse exploration.

Several results connect to the lecture discussion. Model capability dominating team size is consistent with Dwarkesh's own point that more copies of a weaker model cannot solve every problem, and that his population argument depends on capabilities improving at the same time. Adding agents does not add independent ideas unless the structure preserves diversity, and a fixed debate format resulted in the opposite. The self-organization seen in the asynchronous setting is encouraging for productivity. It is also a small example of agents coordinating in ways their designers did not specify, which is relevant when it comes to our understanding of alignment and safety, even in a small setting like this one.

The setting was deliberately small: one task, a maximum of six agents, and 36 five-minute runs per cell. Future experimentation opportunities could look at whether the asynchronous advantage holds with larger teams, longer compute budgets or objectives that are harder to verify than validation loss.

Conclusion and Key Insights

Although Dwarkesh and Daniel addressed different parts of the problem, both treated recursive self-improvement as plausible within the next several years. AI 2040 places fully automated AI R&D around 2030 on the default no-deal path, while under Plan A, AI matching top human researchers arrives in 2035 with a coordinated scaling pause that’s held until 2040. Dwarkesh’s half of the lecture asked what RSI is, what it would take to get there and why it could threaten human control. Daniel’s half asked how such progress might disrupt society and how the available responses differ from one another. Taken together, the two halves move the central question away from whether RSI will arrive and toward whether humans will still be able to steer what follows.

Each speaker approached his half from a distinct vantage point. Dwarkesh’s podcast has placed him in extended conversations with the people building and financing frontier AI, from lab leaders such as Dario Amodei and Ilya Sutskever to executives such as Satya Nadella and, this year, Elon Musk and Jensen Huang, alongside researchers, economists and historians. That breadth gives him an unusually wide view of the industry. His role is closer to that of a synthesizer and broadcaster of information than an insider, reflecting his belief that public understanding and informed decision-making will only grow more important as AI advances. Daniel has spent years thinking with his colleagues at the AI Futures Project about what a transition to advanced AI might look like and how society could prepare for it, work informed by his time at OpenAI and his public decision to leave the company in 2024, and that experience suited the practical, policy-oriented questions his half of the lecture took up.

One thread runs through both halves, and that thread is verifiability. In Dwarkesh’s account, RSI advances fastest where results can be checked, as with held-out loss in pretraining or serving cost in inference, and the class experiment showed agents making real progress on exactly that kind of objective. The experiment also complicated the hope that many copies can pool what they learn, since its agents explored more effectively in isolation than in conversation. Verifiability is weakest, however, where oversight depends on it most. Alignment research lacks an agreed criterion of success even for human researchers, claims about internal acceleration such as Anthropic’s 26% figure are difficult to assess from outside, and Plan A, the option Daniel favored, depends heavily on whether countries can verify one another’s compliance, even though the authors expect verification to be imperfect. This echoes the lecture’s central warning that domains which are easiest to automate are not the easiest ones to oversee.

Read in sequence, the second half also clarifies the stakes of the first. Dwarkesh closed by arguing that continued human control would require AI systems that actively accept human authority despite their capabilities, a problem no one yet knows how to solve. His use of Amdahl’s law showed that the parts of research still dependent on human judgment set the limit on how quickly the whole process can accelerate. Daniel’s comparison of plans can therefore be read as a comparison of how much time each would buy for that unsolved problem. A unilateral slowdown buys only the frontrunner’s lead over its competitors, whereas Plan A would deliberately keep humans in that limiting role for years, slowing development so that oversight could keep pace with capability.

As Boaz noted toward the end of the lecture, Daniel’s risk estimates describe a default path on which nothing changes. AI 2040’s timeline allows more time to depart from that path than AI 2027 assumed, although Daniel himself expects things to move faster than the scenario depicts. Plan A makes that departure concrete, since chip tracking, secure facilities and independent safety evaluation are engineering and policy projects that can begin now. None of this guarantees that humans will remain in control once RSI arrives, but the work that would make that control possible does not have to wait for it.