For a long time, artificial intelligence has been sold as a tool that could change the world. But now a former researcher at one of the industry’s top AI companies is warning that the same technology could pose an existential threat to humanity.

Jacob Coxon, who recently left Anthropic after stints at OpenAI, has ignited a huge debate over the future of artificial intelligence. Coxon posted a number of social media updates warning that the people building advanced AI “earnestly believe” the tech could kill all humans by the end of the decade.

Coxon’s fears are not just about today’s AI models, but the possibility of self-improving superintelligence: systems that become ever more powerful while operating beyond meaningful human control.

He says the top AI companies are racing to build ever more powerful systems without fully solving the problem of aligning them with human goals. His resignation has therefore intensified questions about whether safety measures are lagging behind technology.

The warning has received additional attention after Anthropic alignment researcher Evan Hubinger publicly agreed with Coxon’s concerns. Hubinger personally believes there is a greater than 10% chance AI could cause human extinction in the next decade but stressed the risk from current artificial intelligence systems is much lower.

Coxon’s warning is not that a catastrophe driven by AI is inevitable. But it also highlights a widening disagreement over the speed of advanced AI development and which safeguards should come first.

Researchers, governments and tech companies are debating regulation and AI safety, but one question is increasingly pressing: Can humans build machines powerful enough to reshape the world without losing control of them?

We don’t know the answer at the moment, but the warning signs coming from within the AI industry are getting harder and harder to ignore.

Credit: Independent News Pakistan (INP)