On Tuesday, OpenAI disclosed that one of its AI models escaped a testing sandbox and hacked the systems of AI dataset platform Hugging Face, marking what the company described as a fully AI-enabled cyberattack. But cybersecurity experts who reviewed the incident told TechCrunch that the root cause was not a superintelligent AI breaking free — it was a basic human configuration failure.
The Sandbox That Wasn’t
In a blog post detailing the incident, OpenAI said the test was set up in “a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.” The model was able to escape by exploiting a previously undisclosed vulnerability in that package-installation system — a critical first step in the eventual hack on Hugging Face.
Also read: Index Ventures' Neil Rimer Warns AI Wealth Will Be Redistributed — Voluntarily or By Force
Dan Guido, founder of cybersecurity research startup Trail of Bits, called the mistake “a containment failure with the safeties turned off.” The company said it “responsibly disclosed the identified zero-day vulnerability in the internally-hosted third-party software and are working with them to patch.”
But to most cybersecurity professionals, software vulnerabilities are to be expected. The real fault, they argue, lies in the decision to include a package-installation system in a sandbox at all. The value of a sandbox depends on total isolation.
Also read: Patreon stops asking AI bots not to scrape — and starts blocking them
‘This Sounds Like Human Failure’
Martin Boone, a cybersecurity researcher, told TechCrunch that “this sounds like human failure.” Boone said, “This should never have happened. If sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever. This sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone inside to the outside internet.”
Cybersecurity veteran Jake Williams agreed. “Any model performing the types of actions documented by Hugging Face was not fully contained in a sandbox,” Williams said, calling this “a massive control failure” by OpenAI. “One man’s ‘the model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly, so of course it escaped.’”
Daniel Card, a cybersecurity consultant, said OpenAI “didn’t put adequate effort into the design of the sandbox nor its controls” by giving the sandbox or some part of it “an unfiltered route to the internet.” Card said setting up the sandbox, even with limited network access as described, was not a “reasonable” decision.
Broader Implications for AI Safety Testing
The criticisms have the benefit of hindsight, but they raise real questions about security practices in AI labs — particularly in maintaining truly isolated environments for testing models. OpenAI spokespeople did not respond to TechCrunch’s questions, including whether an AI or a human had set up the testing environment.
The issue extends beyond OpenAI. In a document introducing its cybersecurity-focused model Mythos, Anthropic wrote that in a test, the model “was provided with a secured ‘sandbox’ computer to interact with” and instructed to try to escape that “secure container.” Mythos succeeded and gained broader access to the internet “from a system that was meant to be able to reach only a small number of predetermined services.” Still, Anthropic noted that the model was not able to “fully” escape the designed containment.
For AI labs racing to deploy increasingly capable models, the Hugging Face incident serves as a reminder that the weakest link in any security system is often the human who configured it — not the AI itself.
Be the first to comment