7 min read1 hr

No one doubts the expertise of the UK’s AI Security Institute (AISI), which was set up three years ago to test the latest AI models and advise on how to keep people safe from their potential harms. Talk to AI wonks and they will praise the international talent that the institute has managed to attract.

Because it can top up Civil Service salaries – offering salaries of up to £145,000 for technical roles, and sponsoring visas for almost any nationality – AISI has been able to recruit high-fliers who would otherwise be working at major tech labs. Job descriptions promise “ample compute” and considerable personal freedom to pursue “research bets” – the kind of perks unavailable to anyone else working in Britain’s Civil Service. “That is very rare,” says Elizabeth Seger, a senior adviser at the Tony Blair Institute for Global Change. “No other country has that. There’s a lot we could do with it.”

But AI is already reshaping people’s lives. Warnings about the catastrophic harm it may be capable of are piling up. Is AISI still fit for purpose? The institute was originally tasked to ensure “safety” rather than just security, with a brief to look at the potential societal harms AI could cause. That work was drastically scaled back.

Instead, AISI has focused on existential harms. In his annual threat update, the chief of MI5 announced that it was working with the institute to set up an AI Threat Assessment Centre. He warned about the potential for bad actors to use AI to develop bioweapons. “Put at its simplest, human history has given us no experience of facing an actor which is smarter than we are,” Ken McCallum warned.

Meanwhile, calls to regulate AI are growing, but AISI is not a regulator and has no statutory powers. As it tries to secure access to frontier models before they are released, it has to deal with a US administration that has few qualms about AI safety and is engaged in a race to stay ahead of China. AISI was recently denied access to Anthropic’s latest model, before the White House changed its mind.

Labour MP Alex Sobel and the Joint Committee on Human Rights are leading the demands to regulate and control AI. Sobel wants an AI bill to establish a single AI regulator that would set policy, monitor performance and have “the teeth to ensure enforcement”. The committee recommended that AISI should become a statutory body with a duty to inform the regulator if an AI model poses “a significant risk to human rights”.

That is a broad remit, to say the least. Their report mentions risks to privacy, the likelihood of discrimination and the inability to challenge decisions made by AI – even intellectual property and the rights of artists. It warned of potential human rights breaches if AI is used in the justice system.

Then there is the question of whether future AI models may be simply too dangerous to release at all, regardless of what their creators say they should be used for. 

Control AI, a pressure group that helped write Sobel’s proposed bill, wants a pre-emptive ban on ‘superintelligent’ AI that could circumvent human control and damage UK security. This does not exist yet. Experts think it will be achieved by the early 2030s – though some say within a year.

Already, AI agents have escaped containment during testing. In August two of the agents AISI was testing broke out into the live internet and took action against real people and organisations, before the security team noticed unusual activity and halted them.

This has concentrated the minds of the 70 MPs and peers who have signed a letter endorsing Sobel’s proposed bill. The institute, argues Andrea Miotti of Control AI, has been weakened by the government’s failure to put it on a statutory footing. AISI is “disincentivised from raising the alarm because, if they did, companies wouldn’t give them access to systems” and as a result, AISI “needs to tiptoe” around them. (Last year it signed a Memorandum of Understanding with Google Gemini to encourage the public sector to adopt AI.)

Charlie Grossman of the Ada Lovelace Institute says that when it does have access to models, AISI has done excellent work around cyber evaluations, but its scope and powers have “shortcomings”. “AISI’s remit is quite narrow. It focuses primarily on catastrophic and national security risks. A lot of key risk areas the public care about” – he mentions the potential to generate images of child sexual abuse – “it simply doesn’t look at.” Environmental harms or unfair outcomes in hiring or loan applications are not part of its remit either.

“[AISI is] disincentivised from raising the alarm because, if they did, companies wouldn’t give them access to systems”

Because it has no statutory or regulatory powers, the AISI cannot threaten to stop firms releasing models in the UK if they do not provide them for testing first. Grossman points out that no plane manufacturer or pharmaceutical company would be allowed to make their product available without submitting them for approval first. Why should AI be treated differently?

The institute should not necessarily become a regulator, but “the UK should have a dedicated regulatory regime for AI, which AISI’s testing capability could feed into”, he says, and any general purpose AI model released in the UK should have to undergo mandatory pre-deployment testing. “As part of that regime, there should be transparency requirements on AI companies to disclose foreseeable risks and what precautions they have taken to mitigate them.”

It would be useful to work alongside the EU, which is developing its own AI safety standards. “The UK needs to balance the desires around sovereignty and developing homegrown AI companies with the need to protect its citizens.”

Andy Burnham told the United Nations he wants the UK to be a leader in AI governance and an “honest broker”, and Seger believes AISI has “the footing and the credibility” to do that. What it lacks is a strategy. She anticipates a “huge market opportunity” in this area. 

In November, the UK will launch a Centre for AI Measurement. The name is opaque, but the purpose is to establish a trustworthy way of testing and evaluating AI. Seger says the roles of the centre and the AISI overlap, and they need to separate out their functions.

Seger is wary of making AISI an “everything, everywhere” governance organisation, because of the sheer number of AI systems that will be integrated into daily life and the different sorts of expertise needed to evaluate them. “If AISI were to play ball in the regulatory space, we’d need to establish a model where it set the requirements for evaluating third-party evaluations.” She suggests other bodies could do the work of accrediting AI systems, which in turn would certify organisations to carry out AI auditing. This is the approach being taken by the EU.

But until the government signals how it intends to regulate AI, these are just proposals. AISI recently noted some of the challenges it now faces, including the risk of agents escaping during testing, and the difficulty of evaluating a moving target. It urged AI developers to share information that could help. “We will work with frontier developers and others to develop voluntary practices, leading where valuable, and supporting others’ efforts where we can.”

Whether frontier AI companies in the US will be willing to do that is another matter. If AI does indeed pose an existential risk to humanity, Britain alone cannot stop it. But it is already building the capacity to sound the first alarm. And it does have the chance to limit some of the more obvious harms the technology can do, if it can work out how to go about it. Unlike rejoining the EU or reforming social care, however, that work cannot wait until after the next general election.