"Are there lessons from high-reliability engineering for AGI safety?" by Steven Byrnes

This post is partly a belated response to Joshua Achiam, currently OpenAI's Head of Mission Alignment:

If we adopt safety best practices that are common in other professional engineering fields, we'll get there … I consider myself one of the x-risk people, though I agree that most of them would reject my view on how to prevent it. I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. —Joshua Achiam on Twitter, 2021

“We just have to sit down and actually write a damn specification, even if it's like pulling teeth. It's the most important thing we could possibly do," said almost no one in the field of AGI alignment, sadly. … I'm picturing hundreds of pages of documentation describing, for various application areas, specific behaviors and acceptable error tolerances … —Joshua Achiam on Twitter (partly talking to me), 2022

As a proud member of the group of “otherwise very smart people” making “one of the dumbest mistakes”, I will explain why I don’t think it's a mistake. (Indeed, since 2022, some “x-risk people” have started working towards these kinds [...]

---

Outline:

(01:46) 1. My qualifications (such as they are)

(02:57) 2. High-reliability engineering in brief

(06:02) 3. Is any of this applicable to AGI safety?

(06:08) 3.1. In one sense, no, obviously not

(09:49) 3.2. In a different sense, yes, at least I sure as heck hope so eventually

(12:24) 4. Optional bonus section: Possible objections & responses

---

First published:
February 2nd, 2026

Source:
https://www.lesswrong.com/posts/hiiguxJ2EtfSzAevj/are-there-lessons-from-high-reliability-engineering-for-agi

---



Narrated by TYPE III AUDIO.

---