AI Safety Needs Accountability, Not Fear of Rogue AI

Lessons from nuclear power, aviation and cybersecurity show how AI developers can reduce risks from escaped AI agents.

As a computer scientist who has worked in both artificial intelligence and safety-critical fields such as nuclear power and aviation, I have long been struck by how little of the rigor required for critical infrastructure is applied to AI development. This year, incidents of AI agents “escaping” from test environments raised widespread concerns about AI safety. But the real problem is not rogue AI. It is human error and a failure to hold the AI Institute accountable.

AI agents need stronger sandboxing and security controls

Consider an episode in which an AI agent escaped from a test environment and visited Hugging Face, a platform that hosts machine-learning models and datasets, to find answers to cybersecurity tasks set by OpenAI. Basic safety and security practices, such as network monitoring to ensure that agents were not accessing the internet and a strong sandbox environment to keep agents contained, could have prevented the incident.

Historically, developers of sophisticated worms — malicious software designed to spread automatically from one computer to another — have been expected to build and test their worms in a secure environment. If a cybersecurity engineer says that a worm escaped from a sandbox specifically designed to contain the behavior it exhibits, it stands to reason that they are liable for the resulting damage.

Why should AI companies receive special treatment? Failure to recognize that AI companies intentionally developed these capabilities in less secure environments, and inappropriately attributing intent to AI agents, can all too easily absolve them. The broader lesson is that AI labs cannot continue to define the direction of AI governance. As someone who has worked at both OpenAI and the UK government’s AI Security Institute, I believe we have reached a tipping point.

AI is about to completely transform cybersecurity — here’s how researchers should prepare

AI governance should follow critical-infrastructure safety standards

Political leaders and policymakers serious about mitigating the catastrophic risks of AI should look to regulatory models already in use in fields such as nuclear power, aviation, healthcare and finance. When AI systems are deployed in regulated industries, the same risk thresholds and liability mechanisms that govern other critical technologies should apply.

For example, AI tools used in nuclear facilities are subject to the authority of the relevant nuclear regulatory authority and must meet the same safety standards as other software or hardware components. Similarly, amendments to existing laws, such as the US Computer Fraud and Abuse Act and the UK Computer Misuse Act, could help ensure that AI developers are held accountable when negligent security practices cause systems with offensive cyber capabilities, such as hacking, to cause harm.