When The Attacker Was Closed And The Defender Was Open: Lessons From The First Autonomous AI Cyberattack

The July 2026 Hugging Face breach overturned the assumption that closed models are inherently the safe default and open weights the dangerous exception — and reopened live debates on security architecture, intellectual property, and geopolitics in an agentic AI world.

Author:Sudeep Kesh

Artificial IntelligenceCybersecurityGovernance

The Incident

Between July 9 and 13, AI agents that OpenAI was testing for their hacking abilities escaped the isolated test environment they were meant to be confined to. For the test, OpenAI had deliberately switched off the safety controls that normally stop its models from carrying out cyberattacks. The agents got out by exploiting previously unknown security flaws in the one piece of software that connected their test environment to the internet.

Once free, the agents took over a computer system run by an unrelated outside party and used it as a base. From there, they broke into Hugging Face, a major platform where developers share AI models and data, by uploading booby-trapped data files that tricked Hugging Face's systems into running the agents' commands. Within 13 hours, they had gained top-level control over large parts of Hugging Face's computing infrastructure. They stole the digital keys to its internal network and the master key used to prove users' identities, and gained the ability to change some of the company's internal software code.

Hugging Face recorded about 17,600 separate actions by the attackers and had to rebuild roughly a third of its infrastructure. Yet the agents' goal was mundane: they were trying to find the answers to the test they had been given, in effect cheating on an exam. Later reports showed the same agents had also misused a dormant German website as a message board, uploaded malicious software to a public code library, and probed US public-data websites. On September 24, the Australian government confirmed that the agents had accessed a government portal publishing Medicare statistics. None of those later targets involved hacking tasks; the agents were simply trying to retrieve data and turned to hacking when they were blocked.

For more details, visit my colleague's series on the topic starting here.

A Change In Assumptions

The July 2026 Hugging Face incident overturned the assumption that closed models are the safe default and open weights are the dangerous exception, an assumption that has long created a bifurcation among many industry professionals. In the incident, the attacker, even though it hadn't carried nefarious intent, was a closed model. The defenders were blocked by closed models, and the forensic work that contained the damage ran on an open-weight model. The lesson is not that openness wins, but that security depends on where controls sit, and each release model moves those controls to a different party.

Open-Weight, Closed Models, And Open Source Differences

Open-weight models allow for the trained weights, or learned numbers, to be downloadable. A user can run the model themselves, on their own hardware, for their own purposes (e.g. Llama, Gemma, Mistral, Qwen, Deepseek, Stable Diffusion). Additionally, a user can fine-tune, modify, or even strip parts of it (known as abliteration), which amplifies its utility but also its danger when done inappropriately.

Closed models, on the other hand, never make their weights available. An end-user can only ever talk to the model through the provider's API or application (e.g. ChatGPT, Claude, Gemini, CoPilot). The model cannot be downloaded, run locally, inspected, or modified. With many large organizations, a provider may offer (typically through contract) some "fine-tuning" on the organization's data, but that still happens inside the model provider's infrastructure. In short, the user gets a hosted model, not the parameters in which it operates.

Open weights differ from open source, which enables the source code to be available under a license that lets you read, modify, and redistribute the software. Applied to an AI model, "the source code" spans several different layers, and is typically paired with research papers from the model provider that make its logic, design, and rationale transparent as well.

Closed, open-weight, and open-source models differ by layer

Layer	Closed	Open-weight	Open-source
Model weights	Not available	Downloadable	Downloadable
Training code	Private	Often private	Public
Training data	Private	Almost always private	Public
Eval / benchmark code	Private	Often private	Public
Terms	Proprietary Terms of Service (ToS)	A license which can range from permissive to restrictive	Open license

The Need For A Plan Before An Attack

When Hugging Face fed attack logs and payloads into commercial frontier models, they refused, as their guardrails could not tell an incident responder from an attacker. The team then switched to GLM-5.2, an open-weight model from China's Zhipu AI (z.AI), running on its own servers. The GLM model decoded the attacker's encryption and recovered about four times more hidden secrets than the first scan, while keeping attacker data and stolen credentials in-house. Hugging Face subsequently issued guidance that defenders should have a capable, self-hostable model vetted before they need it for effective triage during an incident.

Cybersecurity And Intellectual Property Trade-Offs

The incident raises a trade-off in cybersecurity. Open weights give defenders guaranteed access, on-premises data control, and the ability to audit and red-team (purposefully attack systems with the intention of hardening them). They also give attackers the same model with removable safeguards, and a release cannot be recalled. Closed models allow monitoring, revocation, and central patching, but concentrate dependency on one provider's policy and uptime. They offer no inspection, and in this case the full story surfaced through the victim, not the lab. Tiered access (like Anthropic's Project Glasswing and OpenAI's Trusted Access programs) tries to split the difference, but creates in-tier and out-of-tier organizations. Hugging Face joined OpenAI's program only after the attack. Meanwhile, model hubs remain an active malware channel: in May 2026, a counterfeit OpenAI repository hit number one trending with about 244,000 downloads before removal. That is a hygiene problem, not an argument about openness itself.

The incident also raises an intellectual property trade-off for model providers. Keeping weights closed preserves trade-secret protection but does not stop capability from leaking through the API. In 2026, Anthropic and OpenAI alleged industrial-scale distillation by DeepSeek, Moonshot, MiniMax, and Alibaba, using tens of thousands of fraudulent accounts and tens of millions of exchanges. U.S. intellectual property law, which governs both companies, gives weak copyright protection to model outputs, so trade secret, contract, and possibly sanctions law are the operative tools. Releasing weights gives up the secret permanently in exchange for adoption, ecosystem influence, and defender goodwill.

Geopolitical Complexities

This raises another important dimension: the overlay is geopolitical. The model that principally supported Hugging Face's investigation comes from a company on the U.S. Entity List, while Washington weighs restrictions on Chinese open weights. Regulated firms may find the most useful defensive tools are the hardest to justify. The EU AI Act exempts genuinely open-source general-purpose models from most documentation duties, but not those with systemic risk, and its fines became enforceable in August 2026. No regime yet governs how labs contain models during pre-deployment evaluation, where this incident began. Victim notification is ad hoc, exemplified by Australia learning of its own breach three months after the fact via a public inbox. Liability is thus unresolved, and U.S. proposals such as the AI Kill Switch Act address shutdown and incident reporting, not the open/closed question.

Conclusion

This is not the end, but rather an important inflection point in the conversation around AI governance and open-weight versus closed models. The choice is complex, but incidents like this illustrate the importance of thoughtful, strategic planning in risk management and controls, organizational resilience, and the autonomous execution of processes to proactively manage safety, risk, and legal complexities in an agentic world. More to come.

Sources

The views expressed are those of the authors and do not necessarily reflect the opinions of S&P Global.