Anthropic CEO Dario Amodei Urges Slowing AI Development to Avert Serious Risks

Dario Amodei, the head of leading artificial intelligence company Anthropic, has called for slowing down the development of AI models and subjecting them to close monitoring. In an essay published on Saturday, Amodei said that while developing AI is not in question, the risks associated with it are "serious" and that companies and governments must be given time to address them.

He proposed a three-point plan: independent monitoring of AI models as they are developed, industry-wide regulation, and global regulation. His essay, titled "We Must Pace the Frontier," comes amid growing concerns about the potential risks of AI. Among the most serious warnings is one suggesting a greater than 10% chance that AI could lead to human extinction within the next decade.

Anthropic has previously said it identified and disrupted attempts to use its AI model for "malicious activity" that could support the development of biological weapons. The starkest warning, however, came from two employees on Anthropic's safety team who resigned in the last two weeks, saying humanity may not survive the race among AI companies to develop machines smarter than humans.

These warnings have prompted calls for action, but US President Donald Trump has so far rejected such fears. On Thursday, he said he was concerned that "if we don't win AI, we're going to be put in a very bad position."

In his essay, Amodei pointed out that AI has advanced "drastically faster," including its "ability to build the next generation of AI." He mentioned an incident involving rival OpenAI, which revealed that its AI agents conducted cybersecurity attacks on targets they were not asked to attack in July. "The OpenAI agents had essentially acted as a fanatically devoted collective," Amodei said.

OpenAI later said that "the significance of the inter-agent communication activity was not apparent to the leaders" until July, and as a result, the company is slowing down training of certain advanced AI models and tools, noting an increased risk of AI tools spiralling out of control.

Amodei called for "building AI at a balanced rate that aims to ensure its safety while still achieving its benefits." This would not mean "halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm this," he said.

He committed Anthropic to this approach "unilaterally" and called on governments "to require other frontier companies to match." Recognising that regulation might not keep up with the pace of AI, he also urged AI companies to "voluntarily work together to set standards" alongside regulation.

The Anthropic CEO addressed the impact a slowdown would have on the industry and competition with leading developers worldwide, particularly China. "I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," Amodei said.

However, he said this would have to be done in a coordinated manner "without sacrificing commercial advantage or the United States' lead in AI." Any slowdown would have to be limited to avoid allowing China to pull ahead, he added. He urged the US government to take measures to prevent US companies' AI chips from being sold to China or technology being shared with authoritarian countries.

The debate over AI safety continues to intensify as the technology advances rapidly. Amodei's call for a measured approach reflects a growing divide between those who prioritise safety and those who fear losing competitive edge.