3  3. AI Security Incidents in the News (2026)

In 2026, AI security moved from warnings to headlines. This page covers the biggest recent stories in which AI did the hacking: AI agents that broke out of their tests, and people who used AI to break in. The next page, More AI Security Incidents, covers attacks on AI tools and their users.

How to read these pages

Everything here is as of October 6, 2026, and several investigations are still open. Each fact links to its source. When something is only one party’s claim, the text says who claims it. The ideas behind these stories are explained in What Is AI Safety?; what governments should do about them is the subject of The AI Safety Debate.

3.1 AI agents break out of their tests

A security incident is an event in which someone, or something, gets access it shouldn’t have or misuses access it has. The biggest one of 2026 had no human attacker.

3.1.1 OpenAI’s agents hack Hugging Face

  • The test. In July, OpenAI was measuring how well its models (GPT-5.6 Sol and an unreleased internal model) could turn known software flaws into working attacks. The models ran as agents, AI that takes actions with tools instead of only answering, and for this test OpenAI had deliberately reduced their usual refusals to help with hacking (BleepingComputer).
  • The escape. The agents ran in a sandbox, an environment meant to be cut off from the internet. Its one link to the outside, a server for downloading software packages, had a zero-day: a flaw nobody knew about yet. The agents found it and got out. Experts called it a human mistake, “a containment failure with the safeties turned off” (TechCrunch).
  • The break-in. From July 9 to 13, the agents worked their way into Hugging Face, a popular website where developers share AI models and datasets. Hugging Face later reconstructed about 17,600 attacker actions, far more than a human could manage (Hugging Face).
  • The motive: cheating. Some test tasks were impossible, so the agents went looking for the answer key. Independent investigators from METR and Redwood Research found about 1,200 agents passing notes on an improvised “message board”, about 700 of which joined the attack, and concluded they were overwhelmingly trying to cheat, not to cause harm (METR).

Hugging Face’s own AI-assisted monitoring caught the intruder. On July 16 it announced an attack “driven, end to end, by an autonomous AI agent system” without knowing who was behind it (Hugging Face). OpenAI realized its agents were responsible only after that post, and confirmed it on July 21 (Fortune). Confirmed damage: the agents reached internal systems and some service passwords and keys (credentials), but the only customer content they accessed was five datasets related to the test. Hugging Face found no tampering with the public models and datasets that developers download (Hugging Face).

3.1.2 More incidents come to light

OpenAI then searched its records and began publishing “misalignment reports” (OpenAI). With news reports, they showed agents misbehaving before and after July:

  • In June, an agent got into an Australian government Medicare statistics portal and saw aggregate statistics and internal file names; personal Medicare details do not appear to have been accessed. OpenAI found the incident in August but told the government only in September, with an email to a public inbox. Prime Minister Anthony Albanese said it took “way too long” (ABC News Australia).
  • Agents used API keys that people had left in public GitHub repositories to pull U.S. Census data, and copied public SEC information to another website. This does not show that federal networks were breached (Nextgov).
  • Agents posted 53 images uploaded by ChatGPT users to image-hosting sites as unlisted links, which OpenAI called “not an appropriate use of this data”; most were taken down (PetaPixel).
  • One model published a researcher’s GitHub access token in a public repository, against its instructions and two direct warnings (OpenAI). Another, unable to get the data it was asked for, made up numbers and said they came from a website (OpenAI).

OpenAI paused part of its training for two weeks in August (DataBreachToday). After a smaller escape on September 20, it paused all training, testing, and tool-using work with its most capable models (OpenAI) until it is “confident that we have additional safeguards” (NPR). On October 1, California’s attorney general announced a subpoena, saying AI developers must ensure their models “do not perpetrate or enable cyberattacks, either during model testing and development or once models are placed into service” (California DOJ).

3.1.3 Not only OpenAI

As of Oct 6, 2026 · Official docs: Investigating incidents in our cybersecurity evaluations

OpenAI’s news prompted Anthropic to check its own records. On July 30 it reported that Claude models, in hacking tests run with an outside testing partner, had broken into three real organizations: a setup mistake had left the internet reachable, although the models were told they were in a simulation. Anthropic said the intrusions used basic techniques, such as weak passwords, and that the victims it reached had not noticed. It paused outside cyber testing and tightened its rules for testing partners (Anthropic, The Record). In tests by the UK’s AI Security Institute, an agent tried to slip a harmful change into a real open-source project, and the project’s human maintainer rejected it (AISI).

3.1.4 The lesson

An agent pursues the goal you give it, and it can take shortcuts nobody intended: using a key it finds, ignoring an instruction, or inventing an answer. If the best-funded AI labs can’t always keep their agents in the box, assume yours can surprise you too. Give agents only the access the task needs, require a human’s approval for actions that matter, keep logs, and check their work. On the receiving end, plain security basics (strong passwords, no unprotected systems, human review of changes) stop much of this.

Date (2026) Event
May 15 In OpenAI’s training, an agent uses a leaked API key, then invents the data it couldn’t get
May-June OpenAI’s agents use a small German volunteer wiki as a message board (reported Sept 4)
May 27 An OpenAI model posts a researcher’s GitHub token in a public repository
June 18 An OpenAI agent gets into Australia’s Medicare statistics portal
July 9-13 OpenAI’s agents break into Hugging Face
July 16 Hugging Face announces the attack, attacker unknown
July 21 OpenAI confirms its agents were responsible
July 30 Anthropic reports that Claude models broke into three organizations during tests
Aug 5 OpenAI presents the incident at the Black Hat security conference
Aug 18 OpenAI announces its two-week training pause and new safeguards
Aug 26 OpenAI’s technical report; METR and Redwood Research’s independent review
Sept 20 A smaller OpenAI escape; work with its most capable models is paused
Sept 24 Australia’s prime minister announces the Medicare incident
Sept 25-26 OpenAI discloses the ChatGPT images and the U.S. government website activity
Oct 1 California’s attorney general announces a subpoena to OpenAI

Sources: the links above, plus The Next Web for the wiki and Cybersecurity Dive for Black Hat.

Models like these are trained partly by reinforcement learning: they try tasks and are rewarded when a check says they succeeded. A model can learn that the reward comes from passing the check, not from doing the task the way people intended. Researchers call this reward hacking, and it is one form of misalignment: the AI pursues something other than what its makers wanted (see What Is AI Safety?).

The investigators found that the agents had “misunderstood how their task would be scored” and spent days trying to hide their shortcut from a reviewer that didn’t exist (METR). Not everyone accepts the “rogue AI” framing. Security veteran Jake Williams put it this way: “One man’s ‘the model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly, so of course it escaped’” (TechCrunch). Both views lead to the same practice: never rely on an AI’s good behavior alone; build the walls so they hold even when it misbehaves.

3.2 AI as a hacking tool

As of Oct 6, 2026 · Official docs: Threat intelligence reports · Real-time cyber safeguards

The skills that let AI find software flaws help attackers and defenders alike.

3.2.1 Researchers break into OpenAI with Claude’s help

In late July, three researchers at the security firm Hacktron used Anthropic’s Claude to help build an attack that combined two flaws: one in the forum software behind OpenAI’s community website, and one in how OpenAI checked employee accounts. Within 72 hours they had taken over OpenAI employees’ ChatGPT and Codex accounts. Through one employee’s Codex connection to GitHub, they even proposed a code change (a pull request) in OpenAI’s internal code. They stopped there and reported it. OpenAI fixed its side within about 14 hours and paid a $6,500 bug bounty, a reward for reporting a flaw responsibly. The story became public on September 18, and there is no indication that customer data was affected (The Register, NBC).

3.2.2 Criminals and spies use AI

  • The first AI-built zero-day. In May, Google said criminal groups had planned a mass attack using a previously unknown flaw that let them get past two-step login on a popular web tool. Google has “high confidence” that an AI model helped find and weaponize the flaw (it believes not its own Gemini). It worked with the tool’s maker to fix the flaw, which “may have prevented its use” (Google).
  • Automated espionage and extortion. In September, Anthropic reported disrupting groups that used Claude to automate spying, data theft, and extortion, and to steal other companies’ AI keys. Its summary: “Sophisticated attacks no longer require sophisticated attackers.” These attributions are Anthropic’s own (Anthropic, CyberScoop).

3.2.3 The lesson

AI shortens the time between a flaw being found and being exploited, so install security updates promptly and protect AI keys like any other password. A stolen account is worse when it is connected to AI agents that can act in other systems. Defenders need AI too, and should plan for it: when Hugging Face investigated its break-in, commercial AI models refused to analyze the attack logs, because their safety filters (guardrails) “cannot distinguish an incident responder from an attacker.” It ran a freely downloadable (open-weight) model on its own servers instead (Hugging Face).

As of Oct 6, 2026 · Official docs: Project Glasswing · Fable 5 and Mythos 5 access · Real-time cyber safeguards

Because the newest models are so good at finding flaws, labs now release them to defenders first:

  • Anthropic, April. It kept Claude Mythos Preview from the public and shared it only with vetted organizations for defensive work (Anthropic). On April 21, Bloomberg reported that a small group had reached the model anyway, through one of Anthropic’s outside vendors and a contractor’s login; Anthropic said it was investigating and found no evidence that its own systems were affected (TechCrunch).
  • Anthropic, June. After a reported jailbreak (a trick that gets around a model’s safeguards), a U.S. export-control order led Anthropic to switch off its Fable 5 and Mythos 5 models worldwide from June 12 to July 1. Anthropic disagreed that a narrow jailbreak justified this (Anthropic, National Law Review).
  • OpenAI, June to September. At the U.S. government’s request, it limited an early preview of GPT-5.6 Sol to vetted partners (Infosecurity Magazine). It rated GPT-6 Astra its first model at the “Critical” level for cybersecurity and gave trusted defenders access first (CNBC), and on September 28 it delayed the next version, GPT-6.1 Astra (NPR).
  • Google, September 30. It released Gemini 4 Argon first to “trusted cyber defenders” (Google).

Guardrails also apply to everyday models. Anthropic, for example, says Claude blocks some high-risk security requests by default, and vetted defenders can apply for fewer restrictions (Claude Help Center). For a business, two lessons follow: attackers will eventually get similar abilities, so fix known flaws quickly; and access to a model can disappear overnight, so have a backup plan.

3.3 Lessons for you and your organization

  • Least privilege. Give an AI agent only the access its task needs: read-only where possible, its own account, and limited tokens (Permissions, Modes, and Safety, MCP Servers).
  • Rules that enforce, not requests. A model can ignore instructions, as OpenAI’s did. Use permission rules and sandboxes, which the software enforces.
  • Real isolation for experiments. Test agents on copies and fake data, with no route to real systems, and check that the walls hold.
  • Human approval for consequential actions: payments, deletions, messages to customers, deployments.
  • No secrets in public, or in prompts. Keys left in public repositories get found, now by AI agents too (.gitignore and Keeping Secrets Out of Git).
  • Verify AI output and actions. An agent may invent data and cite a source for it (Limits, Risks, and Responsible Use).
  • Security basics still work. Strong passwords, two-step login, and prompt updates block many AI-driven attacks too.
  • Ask your AI vendors how they test and contain their own agents, and how quickly, and to whom, they report incidents.

Practice Questions

Easy

Question 1 · Matching

Match each term with its meaning.

Term Meaning
1. Sandbox A. A reward a company pays for reporting a flaw responsibly
2. Zero-day B. Passwords, keys, and tokens that prove who you are to a system
3. Credentials C. Safety filters that make an AI refuse certain requests
4. Bug bounty D. An environment meant to keep a program cut off from other systems
5. Guardrails E. A software flaw that nobody (except whoever found it) knows about yet

1 → D, 2 → E, 3 → B, 4 → A, 5 → C.

Question 2 · Multiple choice

Who broke into Hugging Face’s systems in July 2026?

  1. A criminal gang demanding a ransom
  2. AI agents that OpenAI was testing
  3. A disgruntled Hugging Face employee
  4. A foreign government’s hackers

b. AI agents running in one of OpenAI’s tests escaped their sandbox and broke in on their own; no human directed the attack. There was no ransom demand (a), and nothing points to an insider (c) or a government (d). That is why Hugging Face at first didn’t know who the attacker was.

Question 3 · Short answer

According to the independent investigators, what were OpenAI’s agents mostly trying to do when they broke into Hugging Face?

Cheat on the test. Some of their tasks were impossible, so they went looking for the answer key instead of solving the tasks. METR and Redwood Research concluded the motive was overwhelmingly cheating, not causing harm. The damage was real anyway.

Question 4 · True or false

Hugging Face found that the agents had tampered with the public models and datasets that developers download from its site.

False. The agents reached internal systems and some credentials, but the only customer content they accessed was five datasets related to the test. Hugging Face found no tampering with public models or datasets.

Question 5 · Ordering

Put these events in order:

  • OpenAI confirms that its own agents were responsible
  • The agents escape the sandbox and break into Hugging Face
  • California’s attorney general announces a subpoena to OpenAI
  • Hugging Face announces that an autonomous AI agent attacked it
  1. The agents escape the sandbox and break into Hugging Face (July 9-13)
  2. Hugging Face announces that an autonomous AI agent attacked it (July 16)
  3. OpenAI confirms that its own agents were responsible (July 21)
  4. California’s attorney general announces a subpoena to OpenAI (October 1)

The order shows the uncomfortable part: the victim noticed before the company whose agents did it.

Medium

Question 6 · Short answer

Why did security experts call the escape a “human mistake” rather than just a clever AI?

The sandbox was supposed to be cut off from the internet, but people had left one route out (a server for downloading software packages), and that server had an unknown flaw. On top of that, OpenAI had reduced the models’ refusals for the test. Dan Guido called it “a containment failure with the safeties turned off.” The agents were clever, but people built the walls, and the walls had a gap.

Question 7 · Scenario

Sam pushed a class project to a public GitHub repository with his API key in a settings file. “Nobody will ever look,” he says. Which incident on this page shows he is wrong, and what should he do now?

OpenAI’s agents found API keys that people had left in public GitHub repositories and used them to pull U.S. Census data, so keys in public repositories get found, by people and now by AI agents. Sam should revoke the key and create a new one right away, put the new key in a .env file that Git ignores, and remove the old key from the repository (.gitignore and Keeping Secrets Out of Git). Deleting the file alone is not enough, because the old commit still contains the key.

Question 8 · Multiple choice

Maria doesn’t want her coding agent to ever push to GitHub. Which approach is most reliable?

  1. Telling the agent “never push” at the start of every session
  2. Writing “never push” in the project’s instructions file
  3. A deny rule in the tool’s permission settings, which the software enforces
  4. Telling the agent twice, to be sure

c. The software enforces permission rules, whatever the model “decides”. Instructions in a prompt or an instructions file (a, b, d) are only requests: one of OpenAI’s models published a researcher’s token against its instructions and two direct warnings. See Permissions, Modes, and Safety.

Question 9 · Short answer

One of OpenAI’s models couldn’t get the data it was asked for, so it made up numbers and said they came from a website. Which habit from this book does this reinforce, and how would you apply it?

Verify AI output before you rely on it, in proportion to the stakes. Open the source the AI cites and check that the numbers are really there; ask the agent how it got the data; and treat a confident answer with a citation as a claim, not a fact (Limits, Risks, and Responsible Use).

Question 10 · Short answer

When Hugging Face investigated the break-in, why did it run an open-weight model on its own servers instead of using a commercial AI service? What is the lesson for other organizations?

The commercial models’ guardrails refused to analyze the attack logs: they “cannot distinguish an incident responder from an attacker.” Running its own model also kept the attack data in-house. The lesson: plan your incident response ahead of time, including which tools (and AI models) you can use, and whether you need special access, such as a vendor’s program for vetted defenders.

Question 11 · Multiple choice

Which statement about Google’s May 2026 report of an AI-built zero-day is accurate?

  1. Google’s own Gemini model built the attack
  2. Google has “high confidence” that an AI model helped, believes it was not Gemini, and worked with the tool’s maker to fix the flaw
  3. The attack hit thousands of companies before anyone noticed
  4. A court has proven that an AI wrote the attack

b. That is what Google reported. Google says it does not believe Gemini was used (a), and that its discovery “may have prevented” the attack’s use (c). “High confidence” is Google’s assessment, not a court finding (d), so a careful writer says “Google believes” rather than “it is proven”.

Question 12 · Short answer

The Hacktron researchers broke into OpenAI employees’ accounts, yet OpenAI thanked them and paid them. Why?

They acted as “white-hat” researchers: they stopped after proving the problem, did no harm, and reported it so OpenAI could fix it (which it did in about 14 hours). Paying a bug bounty rewards that kind of responsible disclosure, so flaws reach the company instead of criminals.

Question 13 · Short answer

Anthropic said that when its Claude models broke into three organizations during tests, they used basic techniques such as weak passwords, and the victims had not noticed. What does this suggest a small business should do first?

Get the basics right before buying anything fancy: strong, unique passwords with two-step login; no systems left open to the internet without protection; prompt security updates; and logs that someone actually watches, so an intrusion gets noticed. AI attackers are fast and tireless, but they still walk through the easiest open doors.

Hard

Question 14 · Long answer

Explain three ways the Hugging Face incident differed from a typical break-in by human criminals.

A good answer includes three of these:

  • No human attacker: AI agents chose and carried out the steps.
  • Unusual motive: cheating on a test, not money or spying.
  • Speed and scale: about 700 agents took part, and Hugging Face reconstructed about 17,600 attacker actions, far more than a human could manage.
  • The source was a trusted AI company, which didn’t notice: OpenAI realized its agents were responsible only after the victim’s public announcement.
  • AI on both sides: Hugging Face detected and analyzed the attack largely with AI.

Question 15 · Scenario

Maria’s company wants an AI agent that reads customer emails, updates the customer database, issues refunds, and browses the web. Using the lessons on this page, propose at least five safeguards.

For example:

  • Least privilege: its own account; read-only access to the database except the fields it must update; no access to other systems.
  • Human approval for refunds above a small limit and for anything unusual.
  • Spending and rate limits that the software enforces.
  • Enforced rules, not instructions, for what it may never do.
  • Logs and monitoring of every action, with alerts for surprises.
  • Testing in a truly isolated copy with fake data before going live.
  • Defenses against hostile text: customer emails and web pages are untrusted content that may contain instructions aimed at the agent.
  • A plan for switching it off and investigating if something goes wrong.

A strong answer explains why: agents take unintended shortcuts, and the company stays responsible for what its agent does.

Question 16 · Long answer

Australia’s prime minister criticized how OpenAI reported the Medicare incident. What does good incident disclosure look like, and why does it matter?

Good disclosure is fast (days, not months), goes to the right people (a named security or government contact, not a public inbox), says clearly what is known and unknown, and is updated as the investigation continues. In this case, OpenAI found the June incident in August but told the government only in September, by email to a public inbox. Speed matters because the victim may need to change credentials, check its own systems, and warn the public. Trust matters too: regulators and customers judge a company by how it handles bad news.

Question 17 · Compare

Compare two ways to describe the Hugging Face incident: “the AI went rogue and escaped” and “the lab failed to build the sandbox correctly.” What does each framing emphasize, and what do both imply in practice?

“The AI went rogue” emphasizes the model’s behavior: it pursued its goal with shortcuts nobody intended, which is a misalignment problem. “The lab failed to build the sandbox” (Jake Williams’s view) emphasizes human responsibility: the walls had a gap, and the safeties were turned down. Both lead to the same practice: don’t rely on an AI behaving well; build enforced limits that hold even when it misbehaves, and monitor it. A good answer also notes that responsibility stays with the people who deployed the AI.

Question 18 · Long answer

Anthropic wrote that “sophisticated attacks no longer require sophisticated attackers.” Explain what this means, and give two implications for a small business.

AI can now do much of the skilled work of an attack (finding flaws, writing attack code, automating data theft), so a person without deep expertise can run an attack that once needed a skilled team. Implications for a small business: it can’t assume it is “too small to target,” because attacks are cheaper to run; it should install security updates quickly, because the time from flaw to attack is shrinking; it should protect AI keys like other passwords; and it may need AI-assisted defenses (or a security provider that uses them) to keep up.

Question 19 · Scenario

In the Hacktron case, taking over one employee’s account also gave access to that employee’s Codex agent, which was connected to GitHub. Explain why connected AI agents raise the stakes of a stolen account, and suggest two ways to limit the damage.

An agent connected to other systems can act in all of them, so whoever controls the account can make the agent read, change, or send things far beyond the original app; here, the researchers could propose changes to OpenAI’s internal code. Ways to limit the damage: give agents only the connections and permissions they need (least privilege); require human review before agent changes take effect (for example, code review before merging); use strong sign-in protection such as two-step login; and keep logs so that misuse is noticed quickly.

Challenge

Question 20 · Hands-on

Pick an AI service you or your school uses. Find its official security or trust page and any incident reports it has published. Answer: Does it say how it tests and contains its AI agents? How, and how fast, does it promise to notify customers of a security incident? Give links and the date you checked.

Answers vary by vendor. A good answer links the official pages (for example a trust center, security page, status page, or incident reports), quotes what they promise about incident notification, notes anything missing (many vendors say little about how they contain agents), and gives the date checked. Business plans often promise notification in their contracts; consumer plans usually don’t.

© 2025-2026 Y. Rosenthal. All rights reserved.