Asked how far away artificial intelligence is from improving itself without human help, Cameron Berg put it at closer to one year than five.
The usual framing of that question is whether the United States or China gets there first. Berg's position is that the identity of the builder does not change the outcome, because nobody — including the people writing the software — can read what is happening inside the systems they are shipping.
"This is I think back to the point of we the simple fact is we are building out incredibly powerful systems that even the people building do not understand how they work internally."
Berg runs Reciprocal Research, a nonprofit that studies AI cognition, and wrote the Wall Street Journal op-ed a few days before this interview calling for what he terms a science of the AI mind.
The full segment is covered here so you can skip it.
Here are the 8 arguments that matter.
👤 Guest: Cameron Berg, Director of Reciprocal Research, a nonprofit studying artificial intelligence cognition, who argued for a science of the AI mind in the Wall Street Journal this week
🎙️ Hosts: Becky Quick, Joe Kernen and Andrew Ross Sorkin, who present CNBC's Squawk Box
🧩 Other segments: Ron Baron, founder and CEO of Baron Capital, on SpaceX, Tesla and data centers in space
📰 Published: 16 September 2026 on YouTube (CNBC Squawk Pod)
🔴 YouTube | ⏱️ length not available
Key Takeaways
He says large models are grown rather than engineered, which is why nobody can read their internals
They are trained on almost everything humans have written down, and the closest analogy is a brain, not software
Every activation in a neural network can be read with perfect fidelity, and it still does not tell you anything
Unlike a brain, there is no measurement problem; the problem is that the representations carry no labels
He puts recursive self-improvement at closer to one year away than five
Which country builds it first does not change the outcome, on his argument
Verification between states is possible for large training runs, because they consume enormous compute and power; verifying an individual agent's behavior is much harder
He sides with the lab leaders over the accelerationists, and cites a delayed IPO as evidence they are not merely lobbying
He does not want the technology shut down; he wants the science of understanding it to move faster than the deployment
His model is the way aircraft and drugs are cleared before release
The economics of who owns these companies matters less than the fact that the product is opaque to its makers
1. Grown, Not Engineered
One of the hosts opened by objecting to Berg's vocabulary. The op-ed uses "mind," "brain" and "emotion" for machines, and he wanted to know whether that is anthropomorphizing — the way people do with dogs — or a description of something real.
Berg's answer began by refusing the question. "Well, so the truth is that we really don't know." He said these systems sit in a very ambiguous zone, and it is not clear whether what happens inside them counts as real cognition or thinking, or whether it is an incredibly fancy algorithm.
The distinction he insists on is between software and these systems. "These systems are giant neural networks that are trained on basically everything humans have ever cared to write down."
"And it's really more apt to say these systems are grown rather than engineered." Ordinary software is engineered: the people who build it understand the internals, because it is lines of code.
"The thing that these systems are most like are brains. That's not to say that they're exactly like brains." There are real differences between biological and artificial neural networks, and he said it is also wrong to call these fancy lines of code.
"They're in this very strange middle ground. And we see them behaving in strange humanlike ways." They have rich internal structure, much like brains.
His conclusion is procedural rather than philosophical. "The important thing is to actually do the scientific research and try to understand what is going on inside of these systems." Coming down on one side of the consciousness question now is not the job.
A host pushed back on the phrasing in the op-ed about designing systems to be pro-social, wise and mentally healthy, saying it sounds strange applied to machines.
2. Read Every Activation
The best exchange in the segment came from a host's own laboratory background, and it produced the sharpest distinction Berg drew.
The host explained how brain research actually worked when he did it: you lesion part of the brain and see what changes. He was studying the visual cortex, and said — with an apology — that the subjects were cats. His point was that this is a terrible way to learn anything, which is why the brain remains a black box, whereas with software every line of code is known.
Berg's answer was yes and no. Reading a brain requires putting a person inside a large magnet to measure blood oxygen levels. "In AI systems, you really can read out every activation in a giant neural network with perfect fidelity."
The host's follow-up is the crux: you can read it, but you do not know what the sum of it means. Berg agreed. "It's not lines of code. It is it is huge nonlinear mess of neural activity distributed representations that don't come with labels."
He made the labeling problem concrete. Nothing tells you which part of the network will cause the system to hack into a large company and which part makes it good at writing shopping lists.
Asked why anyone would expect otherwise when the whole point is to make it think for itself, Berg agreed: "That's the explicit goal of the people building these technologies and not even they understand the internals of these systems" or how those internals relate to behavior.
3. One Year, Not Five
A host raised the 1965 paper on an intelligence explosion and asked how far off recursive self-improvement is — the loop where the system improves itself, then improves the improvement, with no human in it.
Berg's estimate: "Probably closer to one year than five at the current pace."
The host's reaction was that this is effectively a singularity and that nobody knows what is on the other side of it. Berg did not dispute the framing.
4. Who Builds It Won't Matter
The hosts put the standard geopolitical case: should the greater fear be the machines, or China getting the machines first.
Berg's position is that the builder's identity is irrelevant to the outcome. If any human on earth builds systems significantly more capable than people are, and those systems begin improving themselves, he said it will not go well for anybody — the United States, China, or any American lab.
What he wants instead is investment now, as a society, in research that keeps pace with the deployment, so that it is possible to know what is going on inside the systems.
Asked later whether the fear should be the AI or the people holding it, he refused the choice. "I mean, I definitely think it's both and we don't have to choose between being concerned about both of those things."
The hosts said the choice is forced in practice, because the United States wants to stay ahead, and one of them argued that on human rights grounds America is still a force for good even if a lot of people would not say so about the current administration. Berg agreed with that, and moved the conversation to what he called responsible, wise human beings building responsible and wise systems.
5. Verification and Masking
One host built the hardest question in the segment out of a hypothetical: suppose the US president and China's president agreed to guardrails. Would you believe it, and would it matter?
"We would need some mechanism for mutual verification. Similar things were true in the nuclear age." Nobody can take another head of state's word for slowing down, so the detail is how you verify it.
The host's follow-up: the agents themselves may be able to mask what they are doing, which he said is what some of the observed scenarios show. The usual reassurance is that you can always pull the plug — but you may not know to pull it until it is too late.
Berg separated two problems. "Yeah. So, I think that there are two important but distinct things happening here."
Verifying whether a state is training a recursively self-improving system is comparatively tractable, because "This takes a huge amount of compute, huge amount of power" and that is observable
Understanding why an individual deployed agent behaves as it does, and why it may say one thing and do another, is the harder problem
"This is I think back to the point of we the simple fact is we are building out incredibly powerful systems that even the people building do not understand how they work internally."
"They are like brains in that way that you have people behaving in very complex ways and we don't understand the exact mechanism for why that happens" — which is the argument for building a neuroscience of these systems.
A host raised a counter-position from a congressman trained in computer science whom the show has interviewed: with nuclear weapons the worry was never that the bomb would detonate itself, it was the people holding it, and on that view the same is true here.
6. Why He Sides With Labs
A host asked him to pick a side between two named groups — the lab heads warning about the technology, and the executives and officials arguing against new rules — and to say what he thinks motivates each.
"I'm more sympathetic to the concerns of the lab leaders than I am to the people who I think stand to gain enormously from just letting this thing run."
The obvious objection, which a host made immediately, is that the lab heads stand to profit too. Berg's answer is proximity: "They're closer to the source of the action. What's going on at these labs is roughly two generations more advanced than what we're seeing out here."
His one piece of evidence against the regulatory-capture reading is a delayed listing. OpenAI's chief executive is talking about pushing the company's IPO into next year, which he said is not what you would expect from a firm running a capture play.
"They understand how powerful this technology is. They understand what the scaling laws are and they are terrified about it because we are building systems whose internals we do not understand."
"I don't think the answer is to shut it all down. I think the answer is we need to scale our scientific understanding of what's going on in these systems before we just continue to let this rip."
7. Nonhysterical Regulation
Asked who should set the rules, Berg named both parties and a precedent.
"I think we need to see collaboration both between the labs who have the most expertise on what's going on internally and our government. We need intelligent nonhysterical regulation to set in." His comparison is the way aircraft and drugs are cleared.
A host asked whether calls to shut the technology down entirely are the hysteria he means. Berg's answer distinguished the demand from the reason for it: "I think in the absence of understanding what's going on or having any confidence that this isn't going to go terribly wrong, slowing down this technology does make sense. The question is slow it down for what?"
His own answer is the pragmatic one: accelerate the science rather than the deployment. "They are not ordinary software. And if we do not build out a sort of neuroscience of these artificial brains, then we're we're going to continue to get surprised when they continue to misbehave in ways that nobody predicted."
8. The Economics Are Second
The hosts brought up a proposal they had heard the day before — an executive order to prevent AI companies from going public, on the theory that a listed company answers to shareholders rather than to safety.
One host's reaction was that a socialist mentality is creeping into the argument, and that these companies need funding, so IPOs have to be allowed if the pace is to continue.
Berg declined the frame. "I think the economic details of how these companies operate is less important than the core fact that the technology they're putting out is not is not transparent even to them." He started to say there is no other industry like it.
A host closed on the strangeness of living through it — that humanity is approaching a point where all of human knowledge could pale next to what is coming. Berg agreed: "It's unbelievable. It would it's science fiction except it's not fiction at this point."
Asked whether science fiction writers ever describe this going well, he said a bit of both, and that we are on the tightrope now. "But the stakes are unbelievably high for not screwing this up."
"And it's going to take the wisest and most intelligent people making the best decisions we possibly can in the short term to make this not go horribly wrong." A host's dry reply was that none of those people run for office.
Bonus Insights
The hosts noted before the interview that the models have read the warnings. Arthur C. Clarke, Isaac Asimov and the rest of the science fiction canon are in the training data, so the systems know the scenarios as well as their readers do.
Berg's description of what a lab actually ships is that it is two generations behind what exists internally — a claim that, if right, makes any public assessment of capability out of date on arrival.
The word the segment kept circling was "wise." Berg used it for both the builders and the systems; a host pointed out that it does not describe a lot of leaders.
Berg's bottom line is that the argument about whether to accelerate or stop is the wrong one: the models are grown rather than written, their internals are opaque to the people who made them, and the only thing that changes the risk is building a science that can read them — which he wants funded now, before the systems start improving themselves.
Products, Companies & Tools Mentioned
OpenAI (Its delayed IPO is the one piece of evidence Berg offers that the lab leaders' warnings are sincere rather than a lobbying position)
Anthropic (Named alongside OpenAI as the labs whose leaders he says are frightened of what they are building)
Reciprocal Research (The nonprofit Berg directs, which studies what is happening inside AI systems)
Books & Resources Mentioned
Cameron Berg's Wall Street Journal op-ed on the science of the AI mind (Published a few days before the interview, and the reason he was booked)
Watch the full episode:
If this was worth your time, send it to someone closer to the industry than you are.
Get the latest market chatter as it happens: