WIRED interviews 10 AI-safety experts on catastrophic risks beyond extinction

Catastrophic AI harm need not involve machines deciding to destroy humanity. In interviews published November 10, WIRED’s Lauren Goode asked 10 AI-safety experts which dangers deserve attention. Their answers included unreliable systems in nuclear operations, an advantage for cyberattackers and easier biological misuse—alongside disputes over extinction forecasts and who should set safety standards.

Different failures need different defenses

Heidy Khlaaf, chief AI scientist at the AI Now Institute and a former OpenAI systems safety engineer, warned about willingness in the nuclear sector to incorporate AI into operations. Her concern is reliability: introducing faulty models into those systems could produce catastrophic failures. She described the issue as urgent, not as a prediction that AI will become conscious or hostile.

Computer scientist Sayash Kapoor identified a different timing problem. He is cautiously optimistic that AI could eventually help defenders improve security faster than attackers exploit weaknesses. For the next few years, however, infrastructure may lack those new defenses, potentially giving attackers the advantage. His warning concerns a gap in adoption, rather than an inevitable harmful effect of smarter AI.

Steph Guerra, a molecular biologist and biosecurity expert at Rand, focused on AI agents entering laboratory workflows. The same assistance that helps legitimate researchers plan and perform tasks could lower barriers to dangerous experiments. She warned that help designing genetic fragments could shorten a virus-production timeline from months to weeks. That is her assessment of a possible misuse pathway, not a newly disclosed biological attack.

Kapoor’s proposed responses were similarly specific: cybersecurity bug bounties reward people for disclosing vulnerabilities, while screening orders for DNA fragments could help address biological misuse. He rejected the idea that improvement in AI’s general abilities is, by itself, a sufficiently precise description of the danger.

It’s better to first specify what downstream risks you’re worried about, because the defenses you come up with will be different.

Sayash Kapoor, speaking to WIRED

Extinction is not the dividing line for every safeguard

The interviewees did not offer a single verdict on existential danger. Stanford professor Rob Reich called human extinction from AI implausible while identifying chemical, biological and radiological threats as major concerns. Guerra, by contrast, described an AI-enabled biological attack as one possible pathway to an existential catastrophe and argued that response infrastructure and safeguards are inadequate.

Yoshua Bengio separated the problem into powerful AI behaving against human intentions, malicious use, concentration of power and unintended consequences. He expressed uncertainty about extinction but argued against taking the chance. He also warned that competition between companies and between the US and China is a struggle for domination at odds with democratic values.

Who decides what is safe enough?

Khlaaf’s criticism extended to the authority to define safety. She argued that established safety engineering requires claims that can be tested and verified, and treats acceptable risk as a democratic question. In her view, large AI labs are instead redefining safety thresholds around hypothetical extinction concerns, claiming decisions that should not belong to engineers alone.

Kapoor described more agreement on governance than on predictions. Among the people he discusses these questions with, he said, there is broad support for company transparency, liability when things go wrong and protection for whistle-blowers. Those measures, he argued, amount to good industrial policy even when people disagree sharply about the scale of coming change.

A development pause remained more contested. Goode wrote that most panelists doubted one would last under companies’ commercial incentives. Bengio nevertheless supported pauses as a way to limit catastrophic damage and give democracy, workers and safety researchers time to adapt. His position makes the disagreement practical as well as predictive: whether slowing development can create useful time for safeguards.