Jacob Coxon Leaves Anthropic and Calls for Limits on AI Capability Gains

Jacob Coxon has resigned from Anthropic and says he is leaving the AI industry, accusing Anthropic and OpenAI of racing toward self-improving AI without acting responsibly. He urged coordinated limits on capability gains and said a temporary ban on improving models may be needed to prevent a global race.

Coxon said he spent three years doing pretraining research at OpenAI and Anthropic. Pretraining is the stage where a model is trained on large amounts of data. His departure is notable less as a dispute over one product than as a challenge to whether competing labs can voluntarily set their own limits.

His concern centers on systems that improve research

Coxon’s warning is aimed at systems that could improve AI research itself, speeding the development of more capable models. He argued that advanced AI could hack computer systems, transform fields quickly, and acquire real-world power and resources. Those are Coxon’s forecasts about future systems, rather than a claim that such capabilities have already been demonstrated.

Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

Jacob Coxon, in an X post

A safety-focused lab still faces the race

Coxon told The Wall Street Journal that Anthropic’s safety work was sincere, but that competition made safety trade-offs difficult to avoid. In his public account, Anthropic understands the risks but keeps competing because it fears another company will behave less responsibly first. That logic makes an individual lab’s restraint difficult: slowing down can look like handing an advantage to a rival.

What Coxon asked researchers to consider

• Whether to participate in reinforcement-learning runs aimed at systems that can improve AI research itself.
• Whether a global race can be prevented without costly action, including a temporary ban on model capability improvements.

The gap between a slowdown and a system for enforcing one

Coxon’s proposed pause is more immediate than a July industry appeal. More than 1,100 AI-company employees, including Anthropic CEO Dario Amodei, signed the Pacing the Frontier letter, which asked governments to develop tools that could slow AI development if it outpaced human oversight. OpenAI and Anthropic endorsed the letter, according to Moneywise.

The difference is consequential. A future government tool would create a route to slow development if oversight falls behind; Coxon is arguing that the industry may need constraints before a global race becomes harder to stop. His resignation does not establish his forecast, but it puts the coordination problem at the center of his break with Anthropic.