Jacob Coxon, a researcher who spent the past three years working on pretraining research at OpenAI and Anthropic, has resigned from Anthropic, accusing both AI companies of racing towards self-improving superintelligence without adequate safeguards.
Coxon announced his resignation in a statement in which he warned that increasingly capable AI systems could soon surpass humans in areas such as cyberattacks, scientific discovery and access to economic and political power.
“Neither company is acting responsibly,” Coxon wrote, arguing that the industry is moving towards what he described as an AI “endgame”.
He said researchers and executives at leading AI companies privately recognise the possibility that advanced AI could pose an existential threat to humanity, despite presenting more measured positions publicly.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon said.
According to him, the concern is not hypothetical because AI systems have already shown rapid progress in hacking, scientific research and other fields, while their capabilities continue to improve.
Coxon drew a distinction between OpenAI and Anthropic in explaining why he believes the race continues despite those fears.
He said many people at OpenAI had not fully internalised what he called the “civilisational stakes” of advanced AI.
At Anthropic, he said, the risks were better understood, but the company remained caught in a race against other developers.
“They believe no one else will act responsibly, so they must do it themselves, despite the risk,” he wrote.
Coxon criticised the idea that private companies should determine how quickly humanity moves towards increasingly autonomous and potentially self-improving AI.
He argued that attempting to accelerate AI alignment, the effort to ensure powerful systems behave in accordance with human interests, requires far greater confidence than currently exists.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he wrote.
Coxon, however, said he remained optimistic that AI companies and governments could still coordinate to slow the race.
He pointed to recent security incidents, including what he described as the Hugging Face attack, as events that could make agreements between major US AI laboratories more realistic.
But he warned that the industry was not currently moving towards preventing a global race for increasingly powerful AI.
Such efforts, he said, could require measures that would carry significant economic costs, including a temporary ban on improving model capabilities.
Coxon also addressed researchers still working inside AI laboratories, asking them to consider what the next several years could look like as companies move towards increasingly autonomous systems.
“Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?” he asked.
He urged researchers to question the argument that advanced AI development will happen regardless of who builds it, and instead use their positions to demand different conditions for further development.
Coxon’s resignation adds to growing public debate over how quickly leading AI companies should pursue increasingly capable systems, particularly as researchers grapple with the possibility that AI development could move faster than society’s ability to control its consequences.
