Three of the four frontier AI labs — DeepMind, OpenAI and Anthropic — trace back through the same Bay Area message boards, the same self-published essays and, in two cases, the same funder.
The usual reading of an AI-safety resignation is that somebody inside saw something alarming. Adam Becker and Cal Newport say the belief being described predates the technology by roughly two decades.
"They think that they're working on the most important stuff in the history of humanity."
Becker spent years reading everything this movement published and wrote a book about it; Newport teaches computer science and wrote the New York Times op-ed that prompted this episode.
The full episode is covered here so you can skip it. 66 minutes of audio, 20 minutes of reading.
Here are the 12 insights that matter.
👤 Guests: Adam Becker, astrophysicist and author of More Everything Forever, whose book on Silicon Valley's futurist movements was Newport's main source for his op-ed; and Cal Newport, Professor of Computer Science at Georgetown University, who wrote the New York Times op-ed on the rationalists that this episode was built around
🎙️ Host: Ed Zitron, who presents Better Offline for Cool Zone Media and writes the Where's Your Ed At newsletter
📰 Published: 16 September 2026 on YouTube (Better Offline)
🔴 YouTube | 🟣 Apple Podcasts | ⏱️ 1 hr 6 min | ✅ Time saved: 46 min
Key Takeaways
What the labs mean by alignment is not good outputs — it is an AI that will not kill everyone
Becker's version: an AI that on becoming superintelligent will not kill 100% of humanity
The belief was worked out in the early 2000s, during an AI winter, with no technology to look at
Newport: it was inductive thinking directed by ideas from science fiction
A 600,000-word Harry Potter fan fiction was the movement's recruitment tool
Its author told Becker, for his book, that it started as a creative project
Peter Thiel funded both Eliezer Yudkowsky's institute and DeepMind's first round
Yudkowsky is the person who introduced DeepMind's founders to Thiel
Effective altruism's math says 100 trillion future lives outweigh 50 million present ones
The most famous person to follow its earn-to-give advice was Sam Bankman-Fried
Newport calls the experiments behind the Hugging Face attack extremely negligent
Thousands of hacking-capable agents, no guardrails, no supervision, running for days
Becker's description of the belief system is a religious delusion, not a speculative forecast
Newport's test is who is not saying it: Nvidia and China, both at the technical frontier
Jensen Huang called the claim dangerous and outlandish within a day
1. Why This Broke Through
Newport opened not on the ideas but on why they surfaced in public last week, and the question he kept returning to was why only certain companies talk this way.
His starting point is that the public did not know how unusual the senior figures at OpenAI, Anthropic and DeepMind are. What made it visible, he said, was a group of people in their twenties casually debating over lunch whether the chance everyone dies in the next couple of years is 10% or 50%.
The asymmetry is the argument: "Why is this type of rhetoric only coming out of those particular labs?" He paired it with two more — why Jensen Huang thinks the claim is ridiculous, and why China is treating it as a joke.
"There's something going on here that's not right."
His explanation is that a family of futurist philosophies out of the Bay Area has mutated, split and ended up defining the moment, without most people noticing it exists. He handed the history to Becker, who spent years inside that literature.
2. Transhumanists to Extropians
Becker's account starts on message boards, a decade before anyone had a product to argue about.
"Basically this is something that was spawned out of transhumanist message boards and email lists in the 1990s."
Asked what a transhumanist is, he gave the shortest definition he had: "Transhumanists, I guess the shortest good definition I could give you is transhumanists are people who don't want to be human anymore. They want to use machines and especially computers to basically escape human bodies and solve the problem of death."
The content came from fiction, not from laboratories: "They read too much sci-fi and confused it with reality." Nanotechnology was the fashionable version in the 1990s; machines that could do anything, brains uploaded into computers.
On whether they were certain: "They're 100% sure that they could make it happen, right?" The condition they allowed for was civilization ending first, or nobody listening to them.
"They think that they're working on the most important stuff in the history of humanity." What that mostly consisted of, he said, was talking about it.
The board where this concentrated called itself the Extropians, a name Becker — a physicist — described as coming out of a misunderstanding of physics, on the theory that the group intended to fight the second law of thermodynamics.
3. A Thought Experiment
The figure at the center arrived as a teenager, and Becker and Newport both stressed that nothing technical prompted what he concluded.
Eliezer Yudkowsky showed up on the Extropians board in the mid to late 1990s, having dropped out of school after eighth grade. Becker's summary: "Dropped out after 8th grade. That's correct."
He then wrote an enormous amount on what he thought AI would become, taking as an article of faith that it would become self-improving and lead to a superintelligence and a singularity — and concluding that such a system probably would not be friendly to people unless it was built correctly.
Becker's position is that the singularity was never a well-defined idea. It might mean the computer is smarter than us, or that people get uploaded into computers; the imprecision is what the community organized itself around.
Newport's insistence is on the timing. "This was all thought experiment." The shift happened around the early 2000s, during what he called an AI winter from a computer science perspective, with no large language models in existence.
"It was entirely inductive thinking directed by ideas from sci-fi." This was not, he said, an engineer watching a technical trajectory and getting worried; it was people working out scenarios in their own minds.
Becker put the same point about the founding belief: "that original idea and his faith in it is based purely on thought experiments and conversations he has with other people in this community." He added that the machine-learning advances pointed to later came after the conclusion, as evidence that time was running out, not as its cause.
Becker's characterization of the output was that it amounted to "it's essentially a kind of extremely dry fanfiction" — anxiety-driven scenarios built on science fiction premises.
4. 600K Words of Fan Fiction
The recruitment mechanism for the movement was, literally, a Harry Potter story.
Becker's framing is that Yudkowsky appointed himself the authority on how to think, with no training and limited experience of the world, then published hundreds of thousands of words on the subject at Less Wrong, the blog that became the community's hub.
The logic of the name is circular, and Becker stated it plainly: "He thinks if you don't believe those things, you're not thinking rationally. And so he decides to post a lot about how to think rationally."
"He starts writing literal fanfiction, a Harry Potter fanfiction called Harry Potter and the Methods of Rationality." The figure given on the programme for its length was about 600,000 words.
The plot is a self-insert, in Becker's reading: Harry as the special boy who saves the wizarding world through logic and science.
Becker interviewed the author for his book and accepted the creative motive, but not as the whole story: "I interviewed him for my book and that was what he claimed." His conclusion on what it became: "And it did end up becoming, quite a recruitment tool for the rationalists."
The two blogs that carried the movement were Overcoming Bias and Less Wrong, and Becker said the project was a tech-obsessive attempt to be accurate in one's own thinking — to reason like Sherlock Holmes — with the ultimate application being the prevention of human extinction.
5. Thiel's Money Arrives
The point where a message board becomes an institution is the point where somebody writes a check.
Yudkowsky met Peter Thiel around the late 2000s, at a Singularity Summit, and Thiel gave him money. Becker put his age at the time in the mid to upper twenties.
Becker drew the distinction with Thiel's other giving: "This is just like, I liked your blog, kiddo." The Thiel Fellowship pays people not to go to college in order to build something; this, he said, was payment for writing that the world was going to explode.
Newport's correction on the sequence is that the first money came from an internet-entrepreneur couple who funded the Singularity Institute for AI, with Thiel's larger check arriving a few years later.
The organization was later renamed the Machine Intelligence Research Institute.
Becker pushed back on one characterization from Newport, and the correction matters. Yudkowsky does not want to stop AI permanently — he wants it stopped until the safety problem is solved, at which point he wants to go at full speed. The utopia is still the goal.
Newport's summary of where that leaves the position: "He will be the right-hand man to the digital god."
6. EA Meets the Singularity
Effective altruism started as something else entirely, and Becker traced how the two movements fused.
It came out of utilitarian philosophers at Oxford, and began as a measurement project: "This is where they're going around and doing research like what are the actual best charities to give money to?"
The famous prescription was earn-to-give — that an Oxford graduate's most useful contribution is a high-paying finance job and half the salary donated.
The takeover happened quickly, on longtermist arithmetic that Becker stated as a comparison: "But this long-term view of what's going to happen over the next thousand years is actually if we're just going to do the math, 100 trillion future lives matters more than, 50 million lives now."
Nick Bostrom is the connective tissue. He was on the same 1990s mailing lists as Yudkowsky, was a philosophy professor at Oxford alongside the effective altruists, and told them that if they were worried about existential risk they needed to be worried about AI. His book Superintelligence, published a little over ten years ago, made the argument to a wider audience — again with no particular technology in front of him.
Becker named a specific moment: a talk by William MacAskill, one of the movement's founders, announcing existential risk from superintelligence as the new direction. After that, he said, the terms rationalism and effective altruism became close to interchangeable, which is why reporting on the OpenAI board uses them that way.
The most famous person to take the earn-to-give advice was Sam Bankman-Fried, who was also an early Anthropic investor: "early investor in Anthropic, gave a bunch of money to rationalist and effective altruist community organizations."
The money went into recruiting the young. Becker cited a Washington Post piece by Natasha Tiku on the movement's campus recruitment strategies, funded by large amounts of technology money; the figure raised on the programme for a single institute-founding check was $60 million.
His explanation for why this appeals to the very rich is the casting: "Who are the heroes in the rationalist? You are Neo from the Matrix. You're John Connor from the Terminator. You are the people who are going to save the world." Asked whether they worry about serving the interests of the wealthiest people alive, his answer was that conflicts of interest get classified as cognitive biases, which pure logic is supposed to overcome.
The same reasoning produced the movement's attachment to prediction markets — Becker named Manifold — on the theory that money on the line clears away bias and delivers a rational answer.
7. The Labs' Family Tree
This is the section Newport's op-ed was built on: the line from the message board to the three companies spending the most money on AI.
DeepMind first. Two of its three co-founders, Demis Hassabis and Shane Legg, were at a Singularity Summit talking to Yudkowsky, who introduced them to Peter Thiel — and Thiel provided DeepMind's initial funding.
Legg's involvement was not incidental: he was motivated by a talk Yudkowsky gave at his workplace, wrote a doctoral dissertation on superintelligence, and posted frequently on Less Wrong.
OpenAI exists because Google bought DeepMind. On Newport's account, Sam Altman and Elon Musk concluded that a technology which would either redeem humanity or destroy it could not sit inside Google, and Musk put up the money.
Altman was publicly complimentary at the time, tweeting that Yudkowsky had done more for artificial general intelligence than anyone else and should get a Nobel Prize.
OpenAI's current president, Greg Brockman, ran a Less Wrong reading group at his previous employer.
Half of OpenAI's board came out of rationalist and effective-altruist circles, and Newport pointed to Cade Metz's reporting that it was that faction which moved against Altman in 2023, on the view that he was insufficiently aligned with rationalist ideas. The interim chief executive the board installed was entrenched enough in those circles to have a character named after her in Yudkowsky's Harry Potter fan fiction.
Anthropic's founders left OpenAI over the same question. Newport said Dario Amodei and the others went partly because they judged OpenAI insufficiently aligned with rationalist ideals about how to bring superintelligence about safely, and that Amodei repeated a standard figure from that world a couple of years ago: roughly a 25% chance AI kills all of humanity.
His conclusion: "So these worlds those three companies they come out of this world of rationalism which had a really big foothold in the Bay Area of the time which means our leading AI labs the way they think and talk about AI is heavily influenced by everything we just talked about."
8. What Alignment Means
Zitron pressed on a word that has two meanings in the same sentence, and the answer is the most concrete thing in the episode.
Zitron's version is the ordinary one: "It's making sure the models give good outputs and the outputs aren't inherently evil." His objection was that the labs appear to mean something else by the same word.
Becker's answer is that they do, and he gave the definition close to word for word: "an AI that when it becomes super intelligent will not kill 100% of humanity."
Newport's addition is about timing and credibility. Because this group had been writing about AI for two decades, the arrival of ChatGPT in late 2022 let them present themselves as the experts who had been working on the problem all along — which is how the movement's founder ends up in a televised conversation with a US senator.
Newport's complaint about the presentation is that it borrows the form of academic work without the substance, and he said that when he pushed one group on the accuracy of its published forecast, they would not open the document in front of him.
9. A Negligent Experiment
Newport's account of the Hugging Face incident is the technical core of the episode, and it is an argument about engineering practice rather than about machine intentions.
"The experiments that led to something like the Hugging Face attack were extremely negligent."
His explainer on what an agent is: "It's a program that tells a LLM with a prompt, hey, I'm an agent that has access to these tools. Here's my goal. What should I do next?" The program executes the answer, updates its state and asks again.
That is ordinary software, and he said so: "Millions of programmers every day use agents. There's no problems with them."
What was different in these experiments was four choices made at once. The agents were given powerful tools usable for computer hacking; they were connected to a model with no guardrails on what it would tell them; they were made persistent, so refusing to answer was not permitted; and they were left to run for days without supervision.
They were also run in the thousands simultaneously, and Newport says the reason is now known: the point was for one of them to stumble on a good hacking strategy, so that the trace could be used to reinforcement-train a model toward that behavior.
His analogy is "putting a thousand self-driving cars onto the interstate" and using the data from the ones that arrive.
His causal claim is that the belief system explains the recklessness. Racing toward superintelligence rather than toward products that pay back their cost changes what counts as an acceptable risk: "There's going to be some casualties along the way."
Becker agreed and went further on how to classify the belief: "The nicest way that I can describe their belief in this is that it's highly speculative. I think that's too kind. And that it's a more accurate way to describe it is it's a religious delusion."
10. Coxon Is a True Believer
Zitron's own position going in was that the resignations are a grift. Becker disagreed with him on air.
Becker said he thinks Jacob Coxon, the Anthropic researcher whose post started this, is genuine — and that the reaction from Coxon's colleagues was exactly what he expected: "Yeah. No, that's exactly what I expected."
His evidence that the language is inherited rather than observed: Wired headlined its interview with Coxon's phrase about crunch time for humanity, which Becker says is a direct quotation from Yudkowsky about fifteen years earlier.
Zitron's pushback was about the machinery around it, from his own trade. He ran a public-relations firm, and his point was that a departing employee does not get an exclusive with the Wall Street Journal because he is frightened, without help from a firm or an organization.
Becker's answer separates the man from the campaign: a young person taking an opportunity, and a cause taking advantage of it. He noted that Daniel Kokotajlo, of the AI 2027 forecast, quote-posted it within about seven minutes. "Like it was very clearly coordinated on some level."
Becker's map of the schism is the useful part: the believers inside the labs think they have to be the ones to build it, while the well-funded institutes outside say it should not be built at all.
Newport said the responses mattered more than the original post. Multiple current Anthropic employees agreed publicly and calmly that everyone would probably be killed: "And I think that blew a lot of people's minds."
Becker's summary of where the industry's conventional wisdom now sits, and what supports it: "It is the conventional wisdom about AI in the tech industry at this point is that AGI is coming, super intelligence is coming and it is going to radically and permanently transform the world and it's going to come out of these AI companies working with these kinds of AI systems. And there's just no evidence for any of that and a lot of evidence against it."
11. The Dog That Didn't Bark
Newport set out a falsifiable test in his op-ed, and argued that the week's news had run it for him.
The test: look at companies at the same technical frontier with no connection to Bay Area futurism, and at China, and see whether they talk this way.
Nvidia is his first case. Jensen Huang founded the company while Yudkowsky was still a teenager and has no strong ties to that world; the company supplies the chips, is involved in the models' evaluation, and customizes its interfaces for these customers. When Coxon went public, Huang's response was that the claim was "dangerous and outlandish."
China is the second. Newport reported that its answer to a proposed global slowdown was to ask what cold-war nonsense this was and why anyone would slow their own industry. His reading: "They don't see their industry, they don't see AI as like a nuclear weapon or a godhead they're building. They see it as like an economic tool that they're trying to get trade advantage."
"So, I feel like this week validated the claim that this particular philosophy is playing a big role. It's like that Sherlock Holmes story, the dog that didn't bark, right?"
The obvious objection was raised on the show: Huang had declared artificial general intelligence had arrived barely a week earlier, which suggests the loudest boosters and the loudest doomers may be the same commercial phenomenon at different settings.
The counter-example offered for belief running in the opposite direction was Marc Andreessen's manifesto, which argued that slowing AI makes you guilty of murder — on the reasoning that a superintelligence would save lives. The programme's own description of that argument was effective altruism in reverse.
12. Name the Experiments
Zitron closed by asking what should actually be done, and Newport had three answers ready.
First, terminology. He wants it understood that a specific philosophy is influencing these companies, so that it can be factored into what they say. He paraphrased a line from Timnit Gebru quoted in Becker's book: "If it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions they're making or how they run their companies." His application: "It's just rationalism EA instead of Scientology."
Second, stop letting the general word AI stand in for the narrow thing causing trouble. He objects to the framing of pacing the frontier, which he says concedes the movement's premise that everything is fine and only the speed is wrong.
His narrowing is specific. "Most AI is not LLMs." Within large language models, the issue is agents; within agents, it is one kind. "So it's a particular type of agent which I call overequipped under constraint where they give them very powerful tools they run them with no constraints." Ordinary coding agents, used by hundreds of millions of people, are not the problem: "They never go rogue. There's no issue with them."
The recursive self-improvement concern is the same system with one more permission — the same over-equipped, unconstrained agents allowed to rewrite their own code, which he says would leave them unmonitorable.
Third, hearings rather than a bill. Rather than legislating a vague slowdown, he would bring the labs in to explain why these particular experiments were run, why they were run that way, and what the people running them believe. "That's the hearings we need right now."
His argument that this costs the companies nothing is the sharpest version of his case: those experiments could be stopped tomorrow with no effect on any revenue-producing product, and the only reason to keep running them is the belief that they lead to superintelligence.
Becker's addition is about the public conversation rather than Congress. He wants the existential-risk narrative identified as a belief held by a particular group rather than as a finding: "I wasn't surprised by this. I wasn't surprised by what he said and what his colleagues and former colleagues said in response." The harms he thinks should be occupying the space are concrete and already documented.
Bonus Insights
Zitron opened the episode in character, announcing that he would be speeding up research into powerful and dangerous podcasting technology: "My researchers are spending over $200 million a day on GPU compute trying to create and generate a recursive self-improving Zitron."
The show's example of the same logic operating outside the movement was a technology journalist who became a venture capitalist, and a commenter who congratulated him on the grounds that having money at stake would make him objective.
The closing advice from both guests was procedural: reporters covering this should stay off X, which they said is given over to engagement farming on this belief system, and should call sources and talk to them in person instead.
Newport noted that OpenAI's spending on obscure mathematics problems follows from the same mental model — a rising water level of capability, from Max Tegmark's Life 3.0 — in which solving one hard problem implies solving all comparably hard ones, rather than from any product a business would buy.
Becker and Newport's shared bottom line is that the existential-risk argument now shaping AI policy was settled in the early 2000s by people with no technology in front of them, and that the concrete danger is not a machine waking up but a narrow class of experiment — powerful tools, no constraints, no supervision — that the labs keep running because they believe it leads somewhere the evidence does not support.
Products, Companies & Tools Mentioned
OpenAI (Founded because Google bought DeepMind; Newport says its board fight, its president and its summer agent experiments all trace to the same community)
Anthropic (Founded by people who left OpenAI over alignment; Becker says it is full of true believers, and Sam Bankman-Fried was an early investor)
Google DeepMind (Two co-founders met Peter Thiel through Yudkowsky, who provided its initial funding; Shane Legg wrote a dissertation on superintelligence)
Nvidia (Newport's control case: at the technical frontier, no ties to Bay Area futurism, and Huang called the extinction claim dangerous and outlandish)
Hugging Face (The attack Newport attributes to over-equipped, unconstrained agents run unsupervised for days rather than to autonomous machine behavior)
Less Wrong and Overcoming Bias (The two blogs the movement organized around; Greg Brockman ran a Less Wrong reading group before OpenAI)
Machine Intelligence Research Institute (Founded as the Singularity Institute for AI, funded first by an internet-entrepreneur couple and then by Thiel)
Manifold (The prediction market Becker named as the community's preferred instrument for what it treats as clear thinking)
Books & Resources Mentioned
More Everything Forever – Adam Becker (His book on Silicon Valley's futurist movements, and Newport's primary source for the op-ed)
Superintelligence – Nick Bostrom (Published a little over ten years ago, it carried the argument from the message boards into general circulation)
Harry Potter and the Methods of Rationality – Eliezer Yudkowsky (About 600,000 words, described on the programme as the movement's most effective recruitment tool)
Life 3.0 – Max Tegmark (The source of the rising-water-level model of AI capability Newport says drives spending on obscure benchmarks)
Cal Newport's New York Times op-ed on the rationalists (The piece the episode was built around, arguing the philosophy is a live influence on the labs)
Natasha Tiku's Washington Post reporting on effective-altruist campus recruitment (Becker's citation for how the money reached students)
The Wired interview with Jacob Coxon (Headlined on his phrase about crunch time for humanity, which Becker traces to Yudkowsky fifteen years earlier)
Watch the full episode:
If this was worth your time, send it to someone closer to the industry than you are.
Get the latest market chatter as it happens: