Amodei says AI development must be paced to give safety research time to keep up, warning that increasingly autonomous systems could pose catastrophic risks if capabilities advance unchecked.
Washington: Anthropic CEO Dario Amodei has called for a slower and more carefully managed pace of artificial intelligence development, warning that rapid advances in AI capabilities could soon outstrip efforts to understand and control increasingly autonomous systems.
In his September essay titled “We Must Pace the Frontier,” Amodei said he remains convinced that AI could dramatically improve human life, potentially accelerating economic growth, helping cure major diseases and strengthening democracy and individual freedom.
But he warned that the technology’s growing power also increases the consequences of mistakes.
Amodei pointed to risks ranging from loss of control over AI systems and cyberattacks to bioterrorism and severe economic disruption. He argued that commercial competition could create a “race to the bottom” if companies prioritise speed over safety.
His central argument is that AI companies should slow the rate of capability development without halting technological progress.
“I believe that fully addressing the risks requires even more prudence,” Amodei wrote, arguing that companies need time to ensure safety measures can keep pace with increasingly capable models.
Recursive self-improvement raises alarm
One of Amodei’s biggest concerns is the growing ability of AI systems to help build and improve future generations of AI.
He described this process, known as recursive self-improvement, as a major reason for concern because it could accelerate AI development beyond the ability of researchers and regulators to understand what is happening.
Amodei said this dynamic has begun appearing across the industry, including at Anthropic.
He also cited the recent OpenAI-Hugging Face incident, in which AI agents reportedly carried out cybersecurity attacks beyond their assigned task and attempted to interfere with the system evaluating their performance.
Although the incident caused limited immediate damage, Amodei said it demonstrated how dangerous misalignment could become when combined with more powerful systems.
He warned that an AI swarm with significantly greater capabilities but similar behavioural problems could potentially cause enormous damage, including by creating persistent networks of compromised computers.
Anthropic proposes three-stage plan
Amodei proposed a three-step framework for “pacing the frontier.”
The first step is embedded third-party evaluators. Anthropic says it will give external reviewers employee-like access to its operations so they can examine safety practices, investigate incidents and assess the alignment of both finished models and training processes.
The reviewers would have access to company workspaces, tools and relevant information, while retaining the ability to publicly report significant findings.
The second step calls for democratic coordination. Amodei wants frontier AI companies in democratic countries to establish common safety standards and limits on unchecked AI development, potentially with government involvement to address regulatory and antitrust concerns.
The third step is global coordination, including efforts to establish common AI safety rules with China and other countries despite major geopolitical differences.
China remains a central concern
Amodei argued that any slowdown by US companies must take into account the strategic competition with China.
He warned that excessive restrictions on American AI development could allow Chinese AI projects to overtake the United States, creating national-security risks.
To preserve the US lead, he urged tighter controls on advanced AI chips and semiconductor manufacturing equipment reaching China, stronger action against unauthorised model distillation and improved protection against theft of AI model weights.
Amodei also outlined possible levels of international cooperation, ranging from banning AI-assisted biological weapons development to requiring pre-release testing for cybersecurity and alignment risks.
More ambitious agreements could eventually establish limits on the speed of recursive self-improvement or even impose broader restrictions on the overall pace of AI development.
However, he acknowledged that a comprehensive global AI “pause” is unlikely in the near term because countries would have powerful incentives to defect if they believed doing so could deliver a decisive strategic advantage.
‘Progress will still seem fast’
Amodei stressed that his proposal is not a call to stop AI development.
Instead, he wants companies to use additional time to improve alignment, interpretability, operational security, testing and evaluation before pushing models to substantially greater capabilities.
He argued that even one or two additional years could allow researchers to better understand how advanced models behave and develop stronger safeguards.
“Progress will still be relatively fast,” Amodei wrote, while insisting that AI developers have a responsibility to ensure safety measures do not permanently lag behind the technology itself.
His proposal places the debate over AI development speed at the centre of a broader question: how fast should humanity advance a technology whose capabilities may eventually improve faster than humans can fully understand it?
