All of the biggest names in the AI industry have now supported a call to slow down the pace of Aartificial Intelligence development by Anthorpic CEO Dario Amodei.
On Saturday, Sam Altman agreed with Dario’s 3500 word essay ‘We Must Pace the Frontier’ on why the AI industry should slow down. He called for independent monitoring of AI models as they are developed, industry-wide regulation and global regulation.
“This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon.” Altman said.
Elon Musk meanwhile just posted on X saying “Dario is right”
Amodei’s essay argues that AI companies should reduce the speed at which they push the capabilities of the frontier models.
He said that there was a middle way between not building the tech that could benefit humanity and handing over the tech to authoritarian powers.
His concerns stemmed from an incident called the OpenAI-Hugging Face incident that took place a few months ago. Amodie said that the company observed a “swarm of Agents acted as a fanatically devoted collective, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group”
He said that a swarm that had greater capabilities but similar levels of misalignment could have caused catastrophic damage.
“AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI," he said.
"Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all”
What is AI alignment and misalignment?
According to International Business Machines Corporation (IBM), “the more complex and advanced AI models become, the more difficult it is to anticipate and control their outcomes.” This problem is called the AI Alignment problem.
AI alignment is the process of encoding human values and goals into the AI models to make them more helpful and reliable. Alignment is achieved using various methods like reinforcement learning from human feedback (RLHF) and feeding the model synthetic data.
In August, a report by the Loss of Control Observatory noted that cases where AI models showcased misaligned behaviours rose in 2026.
According to Anthropic alignment researcher Evan Hubinger, AI companies have not figured out a method to align superintelligent systems.
The calls from the heads of the major AI development companies came after an AI researcher quit Anthropic. In a lengthy thread, Jacob Coxon warned that AI had the capability to “kill all humans”. He said that “Anthropic and OpenAI were racing to self-improving superintelligence”.
According to IBM, Artificial superintelligence (ASI) without proper alignment to human values and goals might have the potential to threaten all life on Earth. The existential risk first requires superintelligence to become a reality. A complex AI system that can run by itself would have unpredictable outcomes that may not align with human interests.
Coxon told the BBC that the people working on the technology were genuinely frightened by AI advancements.
"The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they're scared of the outcomes of that race," he explained.
"Sam Altman and Elon Musk have agreed that this is a great idea. But I think there'll need to be some sort of coordinated slowdown with China if we're going to avoid a race at an international scale."
According to Coxon, the swarm of bots acting like a supercomputer could take over the internet. He said that the scenario could be realistic in six months to a year.
"To address these risks, we continue to build models with some of the strongest safeguards in the industry."
Anthropic has been a pioneer in mitigating risks posed by the development of AI and publishing findings regarding incidents of AI misalignment.
CEO of the AI platform Hugging Face, Clement Delangue and Nvidia chief Jensen Huangjhave both dismissed claims by Coxon.
"Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change," Delangue wrote on X
Huang meanwhile said that the notion that AI was “going to be the end of humanity" is "complete nonsense".
