Why the intelligence explosion can't happen inside a data centre | Tom Reed

Why the intelligence explosion can't happen inside a data centre | Tom Reed

Author: The 80,000 Hours team September 10, 2026 Duration: 22:10

AI systems are starting to build themselves. Because each generation of model will be better at building its successor than the last, it seems plausible that the full automation of AI R&D could rapidly lead to an exponential growth in overall AI capabilities. A natural inference is that domain-general superintelligence arrives shortly after AI research is automated.

Host Tom Reed does not think this will happen.

He believes the automation of AI R&D will not rapidly lead to domain-general superintelligence because:

  1. It’s impossible to get good at most things without practice.
  2. AI companies lack the data their models would need to practice most things.
  3. This can’t be fixed with “sample efficiency.” In most cases, the relevant data doesn’t exist at all.
  4. This also can’t be fixed with simulations or synthetic data.
  5. This means that the relevant data for superintelligence in most non-coding domains will only become available through deployment of AI models throughout the economy.

The singularity, therefore, will be bottlenecked on signal. The output of the R&D produced by an isolated data centre of geniuses would be a mere “Goodhart Singularity”:

Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.


An isolated AI improving itself against benchmarks would only appear to be approaching superintelligence, while actually optimising for eval performance that fails to generalise beyond the lab.

This suggests that the automation of AI research will not rapidly produce superintelligent capabilities in other domains — their arrival will largely be a function of deployment and data collection in the real world. AI models need real-world deployment for the same reason the body needs pain and corporations need profit: signal is sovereign.

This essay takes each of the above points in turn.

Learn more, video, and full transcript: https://80k.info/goodhart

“The Goodhart Singularity” originally appeared on Tom’s Substack in May 2026, and this narration was recorded on August 26, 2026.

Chapters:

  • Introduction (00:00:00)
  • Practice makes perfect (00:05:05)
  • Good data is hard to find (00:08:22)
  • Simulation is shallow (00:13:43)
  • What a Goodhart Singularity looks like (00:19:04)

Our production team includes:

  • Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour
  • Producers: Elizabeth Cox and Nick Stockton
  • Coordination and support: Katy Moore and Lou Moran
  • Camera operator: Dominic Armstrong

The 80,000 Hours Podcast, from The 80,000 Hours team, digs into the complex and often overlooked questions surrounding how we can best use our careers to tackle the world's most pressing problems. While artificial intelligence is a recurring and critical theme, framing some of the most important conversations you won't hear elsewhere, the discussions range far wider into the intersections of technology, policy, philosophy, and global priorities. Hosts Rob Wiblin, Luisa Rodriguez, and Zershaaneh Qureshi guide in-depth interviews with researchers, policymakers, and practitioners, breaking down daunting ideas into actionable insights. You'll hear nuanced analyses of career paths, ethical dilemmas in emerging tech, and evidence-based strategies for creating a positive impact. This isn't about quick tips; it's about deep, substantive exploration of how specific choices and systemic changes can lead to a better future. The podcast lives in the Education and Technology categories because it fundamentally aims to equip listeners with the knowledge and perspective to navigate a rapidly changing world thoughtfully. Each episode is built on rigorous research, challenging assumptions while maintaining a conversational and accessible tone. Tune in for a consistently engaging and intellectually honest look at the forces shaping our century and the practical steps individuals can take within their own 80,000-hour working lives to make a meaningful difference.
Author: Language: English Episodes: 50

80,000 Hours Podcast
Podcast Episodes
What the hell happened with AGI timelines in 2025? [not-audio_url] [/not-audio_url]

Duration: 25:39
In early 2025, after OpenAI put out the first-ever reasoning models — o1 and o3 — short timelines to transformative artificial general intelligence swept the AI world. But then, in the second half of 2025, sentiment swun…
#234 – David Duvenaud on why 'aligned AI' would still kill democracy [not-audio_url] [/not-audio_url]

Duration: 2:32:50
Democracy might be a brief historical blip. That’s the unsettling thesis of a recent paper, which argues AI that can do all the work a human can do inevitably leads to the “gradual disempowerment” of humanity.For most of…
#233 – James Smith on how to prevent a mirror life catastrophe [not-audio_url] [/not-audio_url]

Duration: 2:10:33
When James Smith first heard about mirror bacteria, he was sceptical. But within two weeks, he’d dropped everything to work on it full time, considering it the worst biothreat that he’d seen described. What convinced him…
2025 Highlight-o-thon: Oops! All Bests [not-audio_url] [/not-audio_url]

Duration: 1:40:17
It’s that magical time of year once again — highlightapalooza! Stick around for one top bit from each episode we recorded this year, including:Kyle Fish explaining how Anthropic’s AI Claude descends into spiritual woo wh…