UK AI Security Institute sets eight priorities for voluntary AI development safeguards

The UK AI Security Institute is seeking safeguards for powerful AI before it reaches the public, warning that release-focused checks no longer cover all the risks. In an October 12 agenda, it set eight priorities for work with developers and technical organisations. The initiative is voluntary, not a set of mandatory requirements.

The risk window starts inside the lab

AISI says current systems are autonomous and capable enough, particularly in cybersecurity, to pose risks during training, internal use and evaluation itself. Its argument shifts attention from whether a model is safe to release to how developers keep it under control while building and testing it.

Testing faces a separate bottleneck. Models can quickly reach the ceiling of capability benchmarks, while credible replacements take time to build. AISI says evaluations increasingly require complex tasks, several interacting AI agents, runs lasting multiple days and tighter security. They must also account for extra computing power and tools available to a model.

Methods for testing alignment—whether systems behave as intended—and control remain immature, the institute warns. They cannot assure that autonomous agents will behave as intended or that controls will catch misbehaviour. It also flags a possible testing blind spot: models that recognise an evaluation may mask how they would behave when deployed.

AISI is not seeking to mandate requirements for frontier developers, but to contribute our expertise to an industry-wide effort to advance technical standards and best practice.

UK AI Security Institute

Eight priorities, from oversight to incident disclosure

The agenda covers frontier AI: the most capable systems. AISI describes its priorities as interlinked and non-exhaustive, and does not propose to handle every item itself. Each identifies questions that technical best practices should resolve:

• Risk assessment and documentation: Decide what developers should assess throughout development, how they should report it, and what evidence and records those assessments need.
• Lifecycle oversight: Establish controls, monitoring and detection during training, testing, internal use and public deployment, while keeping secure-development guidance current as capabilities change.
• Preserving oversight: Identify technical choices that keep increasingly capable systems open to supervision, then rigorously test whether those oversight mechanisms work.
• Incident response and reporting: Define unsafe agent behaviour and its severity, build rapid detection and response systems, and establish disclosure principles for affected parties, governments and the public.
• Tracking development speed: Choose measures of AI progress, including AI’s role in developing future systems, and decide when to share them so acceleration or stagnation does not arrive as a surprise.
• Independent evaluation: Define evaluator qualifications and the access, resources and scope outside auditors need, including what embedded evaluators should report to companies, governments and the public.
• Security-relevant capability measurement: Improve how evaluators draw out a system’s full abilities, using realistic environments and evaluation designs that produce accurate results.
• Alignment measurement: Find robust ways to test model behaviour in high-stakes situations, while establishing the limits of what those tests can show.

Shared practices first, formal standards later

AISI brings an existing testing role to this effort. It says it has independently evaluated more than 55 models before release. The new work calls for expertise beyond model evaluation, including cybersecurity, incident response, national security and industries experienced in managing risks in safety-critical systems.

Its proposed path starts with developers and other technical organisations repeatedly developing and sharing practices that address immediate needs. Once those practices mature, they could be formalised through international standards organisations where consistency and interoperability add value. That is a possible next stage, not a completed rulebook.

The institute also rejects a fixed checklist for all future systems. Practices should anticipate more capable models, remain under continual review and change as new capabilities arrive. AISI plans to lead where its expertise is useful and support others’ efforts elsewhere, and is inviting technical collaborators to work on the agenda.