Hinton and Bengio Urge Government Oversight of a Possible AI Research Surge

AI is not yet speeding up its own development enough to trigger the surge described in a new report. But its authors, including Geoffrey Hinton and Yoshua Bengio, want governments to prepare now. They warn that automated AI research could eventually compress years of progress into months, leaving far less time to respond.

How the acceleration could start

The paper, titled “What if automating AI R&D triggers an intelligence explosion,” has more than 20 authors. Alongside Hinton and Bengio are Anthropic co-founder Jack Clark and OpenAI chief scientist Jakub Pachocki, according to The Guardian. The authors call an “intelligence explosion” a dramatic acceleration in AI progress driven by AI itself. It is a possible outcome, not a development they say has begun.

Their proposed mechanism is a feedback loop. AI systems would do more of the work of designing and improving AI; the improved systems could then take on still more research. The authors consider this a likely route to rapid acceleration because AI already contributes to its own development, and improved systems can be deployed quickly once built. Anthropic says AI produces 80% of its code, though that company figure does not show that the feedback loop has reached the point the paper describes.

The risks move at different speeds

The paper describes several ways a sudden leap could strain existing safeguards. More capable systems might enable biological or cyber threats faster than defenses develop. If humans play a smaller role in AI research, they could lose opportunities to control the systems doing it. A state with a modest technological lead could also turn it into a decisive advantage, the authors warn. These are possible consequences, not outcomes the report says are inevitable.

The same acceleration could produce medical breakthroughs and other technological advances. Nor would every discovery immediately change the world outside a lab: a new technology might still require special materials, supply chains or regulatory approval. AI might also help develop countermeasures faster. The authors acknowledge that the overall effects remain uncertain, even as they argue that a short response window warrants preparation.

Once an intelligence explosion begins, the window for action may close.

Authors of “What if automating AI R&D triggers an intelligence explosion”

What the authors want governments to do

Their first priority is visibility. They recommend requiring AI developers to report progress on AI-related research and development, including through independent auditors embedded at companies. That would give governments a way to follow how much research AI systems can do, rather than waiting for a sudden jump in capability to become apparent.

The report also proposes ways to slow development if needed, including limiting how much an AI system can improve over a given period and working with data centers to pause certain AI research projects. It calls for automated research systems to be isolated so they cannot escape human control, alongside emergency plans for different scenarios. Those proposals address different problems: spotting acceleration, retaining a way to intervene and preparing for what might follow.

A forecast, not a deadline

The authors say AI could fully automate by 2028 some research projects that would take people months. That is a forecast about particular projects, not a prediction that an intelligence explosion will happen in 2028. They also say the productivity gains from AI research automation have not yet reached the level needed to trigger such an explosion, although they believe gains from newer systems may be approaching it.

That distinction defines the policy choice. Acting now would mean putting reporting, oversight and contingency plans in place for a threshold that remains uncertain. Waiting would avoid imposing development limits based on a scenario that may not unfold, but it could leave governments with less time if automated research starts advancing much faster. The paper argues for preparation before that choice becomes urgent.