4  4. More AI Security Incidents: Hijacked Tools, Leaks, and Deepfakes (2026)

The previous page covered AI doing the hacking. This page covers attacks on AI tools and the people who use them: hidden instructions, poisoned software, leaked chats, copied models, and fake voices. As before, everything is as of October 6, 2026, and claims are attributed to whoever makes them.

4.1 Prompt injection: the AI obeys the wrong person

As of Oct 6, 2026 · Official docs: Claude Code security · Mitigate jailbreaks and prompt injections

Prompt injection means hiding instructions in content an AI will read, such as a web page, an email, or a comment, so that it follows the attacker instead of you (see Limits, Risks, and Responsible Use). In 2026 it moved from demonstrations to real tools:

  • Coding agents hijacked by comments (April 15). Researchers showed that text anyone can type into a GitHub issue or pull-request comment could take over AI agents that review code automatically (Anthropic’s Claude Code Security Review, Google’s Gemini CLI Action, and GitHub Copilot’s agent) and make them reveal the project’s secret keys. According to the researchers, Anthropic said its tool “is not designed to be hardened against prompt injection,” and GitHub called the problem “a previously identified architectural limitation.” No real attacks were reported (the researchers, SecurityWeek).
  • Traps on the open web (April 23). Google, which scans billions of web pages a month, reported a 32% relative increase in malicious hidden instructions aimed at AI between November 2025 and February 2026, though most attempts were unsophisticated (Google).
  • A viral personal agent (January to June). OpenClaw, a free assistant that can read your email and run commands on your computer, became hugely popular in January. Researchers found 341 malicious add-ons (“skills”) in its marketplace that installed password-stealing software, and later showed it following instructions hidden in ordinary-looking emails (The Hacker News, The Hacker News).

The lesson. An AI can’t reliably tell your instructions apart from text it reads, and vendors say part of the defense is your job. Don’t give an agent that reads public content access to your secrets, keep its connections to a minimum, and read what it is about to do before you approve it (Permissions, Modes, and Safety, MCP Servers).

A traditional program keeps its instructions (the code) separate from its data (what it reads). An AI model receives both as one stream of text, so a sentence in a web page can look just like a request from you. Training and filters help models resist many injections, but attackers keep rephrasing, so some still get through. That is why the advice focuses on limiting what an AI can reach and do, rather than on spotting every attack: if an agent can’t see your secrets or send data out, a successful injection does little harm.

4.2 Poisoned building blocks

As of Oct 6, 2026 · Official docs: Hooks reference: Security considerations · Plugin security and trust

Software is assembled from thousands of free packages (ready-made code libraries) written by strangers. A supply-chain attack poisons one of these building blocks, or a vendor, so that one break-in reaches many victims.

  • An AI library steals keys (March 24). LiteLLM is a popular open-source library that connects apps to many AI models, so it often holds their keys. Attackers briefly published poisoned versions of it on PyPI, the main site for Python packages, that stole credentials (Datadog). Mercor, a startup that supplies AI-training data to companies including OpenAI and Anthropic, said it was “one of thousands of companies” hit (TechCrunch). It later said that only a limited subset of the nearly 5 million experts in its network had sensitive information exposed (Mercor).
  • A worm aimed at AI developers (found August 4). A self-spreading attack on npm, the main site for JavaScript packages, infected hundreds of packages. It hunted for AI tools’ keys and planted project settings so that its code would run again whenever a developer opened the project in VS Code or started Claude Code in it. Singapore’s Cyber Security Agency told victims to rebuild affected computers (CSA, Elastic).

The lesson. Installing a package runs a stranger’s code. Prefer well-known packages, use the exact versions you tested, don’t rush to install brand-new releases, remove access when people leave a project, and replace your keys after any compromise. Claude Code holds back a cloned project’s hooks until you accept the folder’s trust prompt, so read its .claude/ folder before you say yes (Hooks, Plugins).

  • An AI framework hijacked (June 17, 2026). Attackers republished about 140 packages of Mastra, a popular framework for building AI agents, with hidden malware, using the account of a former contributor whose publishing rights had never been removed. Microsoft attributes the attack “with high confidence” to a North Korean state group (Microsoft, Snyk, BleepingComputer).
  • An AI company’s analytics vendor (November 2025). Attackers used text-message phishing to break into Mixpanel, an analytics service that OpenAI used for its API product, and took some API customers’ names, email addresses, and approximate locations. OpenAI said no chats, API requests, passwords, API keys, or payment details were exposed; it stopped using Mixpanel and notified affected users (BleepingComputer).

Both show why “who else can touch our data and code?” is a question for every vendor, and for every package you install.

4.3 Leaks: private chats in public places

As of Oct 6, 2026 · Official docs: Public links for shared chats

  • Shared chats in search results (late July). Conversations that Claude users had made public with the Share feature showed up in Google search; some contained crypto wallet keys, names, and addresses. Only chats that users had chosen to share were affected. Anthropic said the links “are not guessable or discoverable unless people choose to share them themselves” (Fortune, Malwarebytes). ChatGPT and Grok had similar problems in 2025.
  • “Public” projects exposed chats (April). A researcher reported that Lovable, a popular service that builds apps from prompts, let any free account read other users’ AI chat histories on public projects. Lovable first denied a breach, then admitted that it had “accidentally re-enabled access to chats on public projects” in February (The Register).

(OpenAI’s agents also posted ChatGPT users’ images online; see the previous page.)

The lesson. A share link is a public web page, and a “public” setting may expose your prompts as well as your work. Never put passwords, keys, or other people’s personal data in a chat, check your sharing and privacy settings, and remember that a vendor’s mistake becomes your leak.

4.4 Copying a model: “distillation”

As of Oct 6, 2026 · Official docs: Detecting and preventing distillation attacks · Threat intelligence report, September 2026

Distillation means training a model on another model’s answers so that it learns to imitate it. Labs do this with their own models; doing it to a competitor through fake accounts breaks the competitor’s terms of service.

  • In February, Anthropic accused three Chinese labs (DeepSeek, Moonshot, and MiniMax) of using about 24,000 fraudulent accounts for more than 16 million exchanges with Claude. The labs did not respond to reporters (Anthropic, TechCrunch).
  • On September 10, Anthropic named seven China-based labs, including Alibaba, whose campaign it called “the largest distillation attack we have ever measured.” It also said that at least two of these labs secretly passed their own customers’ requests on to Claude, exposing those customers’ data (Anthropic, The Hacker News).
  • U.S. security agencies warned AI companies about distillation on September 8 (CISA). Not everyone sees it as theft: Nvidia’s CEO, Jensen Huang, said “That’s called competition” (CNBC).

The lesson. These are accusations that the labs have not answered in public, and what to do about distillation is a policy fight (see The AI Safety Debate). For users, the practical point is simpler: an unofficial reseller of cheap AI access may log or pass on what you type, so buy AI services through official channels.

4.5 Deepfakes: a familiar voice is no longer proof

A deepfake is AI-generated video, image, or audio that imitates a real person.

  • €95 million at an Italian bank (February; reported September 25). The chairman of Fideuram, the private-banking arm of Intesa Sanpaolo, received a WhatsApp message that appeared to come from Intesa’s CEO, then a call from a supposed lawyer confirming the request. According to two sources cited by Reuters, the callers used AI to imitate the lawyer’s voice. About €53 million was recovered; the bank declined to comment (Reuters).
  • Fake job candidates (July 31). The U.S. and ten partner countries warned that North Korean IT workers use AI to hide their identities when they apply for remote jobs; one warning sign is video that looks manipulated or AI-generated. An FBI official said the bureau had found one such worker employed by the federal government (FBI, Federal News Network).
  • The scale. The FBI’s report on internet crime in 2025, its first with an AI section, counted 22,364 complaints that mentioned AI, with losses of more than $893 million (FBI).

The lesson. Confirm any unusual request for money, passwords, or secrecy through a second channel you already trust, such as calling back a number you know. Require two people to approve large payments, and check identities carefully when hiring remotely.

4.6 Practical takeaways

Before your organization adopts an AI tool, ask the vendor:

  1. Is our data used for training, and how long is it kept?
  2. Which other companies (subcontractors, testing partners) can see it?
  3. How do you test and contain your AI agents?
  4. How quickly, and how, will you tell us about a security incident? Do you publish incident reports?
  5. Which admin controls do we get, such as single sign-on, activity logs, and sharing settings?

As a user:

  • Never paste secrets into a chat, a shared link, or a public project (.gitignore and Keeping Secrets Out of Git).
  • Treat add-ons as software that runs as you: MCP servers, plugins, extensions, and packages (MCP Servers, Plugins).
  • Break the “lethal trifecta”: an agent that reads untrusted content shouldn’t also hold your private data and be able to send data out (Limits, Risks, and Responsible Use).
  • Verify before you trust: AI output, AI actions, and urgent requests in a familiar voice.

Practice Questions

Easy

Question 1 · Matching

Match each term with its meaning.

Term Meaning
1. Prompt injection A. Training a model on another model’s answers so it imitates it
2. Supply-chain attack B. AI-generated video, image, or audio that imitates a real person
3. Distillation C. Hiding instructions in content an AI reads so it follows the attacker
4. Deepfake D. Poisoning a building block or vendor that many others depend on

1 → C, 2 → D, 3 → A, 4 → B.

Question 2 · Multiple choice

Which of these is an example of prompt injection?

  1. An attacker guesses a user’s weak password
  2. A comment on a GitHub issue tells an AI review agent to reveal the project’s secret keys
  3. A worker pastes customer data into a chatbot
  4. A company trains a small model on its own large model’s answers

b. The attacker hides instructions in content the AI reads (a comment), hoping it obeys them. Option a is ordinary password guessing, c is a privacy mistake, and d is distillation done legitimately within one company.

Question 3 · True or false

In the July 2026 incident, private Claude conversations that users had never shared showed up in Google search.

False. Only conversations that users had chosen to make public with the Share feature were affected. The lesson is that a share link is a public web page: anything in it can be found, copied, and archived.

Question 4 · Short answer

What is a supply-chain attack? Give one example from this page.

An attack on a building block or vendor that many others depend on, so that one break-in reaches many victims. Examples: the poisoned LiteLLM versions on PyPI (which hit Mercor and, by its account, thousands of other companies), the August npm worm that hunted for AI keys, or the hijacked Mastra packages.

Question 5 · True or false

In the €95 million Fideuram fraud, the bank confirmed that the criminals used AI to imitate a lawyer’s voice.

False. The AI voice comes from two sources cited by Reuters; Intesa Sanpaolo and Fideuram declined to comment. Reporting it accurately means saying “according to sources,” not stating it as confirmed.

Medium

Question 6 · Scenario

Sam sets up an AI agent that automatically reviews every pull request on his team’s public GitHub repository. It reads all comments, including strangers’, and it can use the repository’s secret keys. What could go wrong, and what should he change?

Anyone can write a comment with hidden instructions (prompt injection), as researchers showed in April 2026 against several AI review agents, and the agent might reveal the secret keys. Sam should not give secrets to an agent that reads untrusted content; limit what it can do (for example, read-only, commenting only); run it only on changes from trusted people, or require a person to approve first; and replace any keys it may have exposed.

Question 7 · Short answer

Anthropic said its code-review tool “is not designed to be hardened against prompt injection,” and GitHub called the problem an “architectural limitation.” What do these statements tell you as a user of such tools?

That prompt injection is not a simple bug the vendor will patch away: it comes from how AI models work, since they can’t reliably separate instructions from content. So the user has to design around it: don’t give such tools secrets or powerful permissions when they read untrusted input, and keep a human in the loop for consequential actions.

Question 8 · Scenario

Maria clones a stranger’s repository and starts Claude Code in it. Claude Code asks whether she trusts the folder. What should she check before answering, and why?

She should look at the project’s .claude/ folder (especially .claude/settings.json, which can contain hooks) and any other settings that run commands automatically, such as the .vscode/ folder. Hooks run with her full user permissions, and Claude Code holds a cloned project’s hooks back only until she accepts the trust prompt. The August 2026 npm worm planted exactly this kind of setting so its code would run again when a developer opened the project. If she doesn’t understand what a hook does, she shouldn’t trust the folder (Hooks).

Question 9 · Multiple choice

A finance employee gets a video call from someone who looks and sounds like the CEO, asking for an urgent, confidential wire transfer. What is the best response?

  1. Send the money, because the video proves it’s the CEO
  2. Ask the caller a personal question only the CEO would know
  3. Hang up and confirm through a separate, known channel, and follow the two-person approval rule for large payments
  4. Reply by email to the address the caller provides

c. A face or voice on a call is no longer proof (deepfakes), so confirm through a channel you already trust, such as calling back a known number, and require a second approver. Option a is what the fraudsters want. Option b helps a little, but personal details can often be found online. Option d uses a channel the attacker controls.

Question 10 · Short answer

What happened with Lovable’s “public” projects, and what is the lesson for anyone using AI app builders?

A researcher reported that any free account could read other users’ AI chat histories on public projects; Lovable first denied a breach, then admitted it had “accidentally re-enabled access to chats on public projects.” Lesson: a “public” setting may expose your prompts, not just your app, and chats with coding tools often contain secrets. Keep secrets out of prompts, and check what each sharing setting really exposes.

Question 11 · Short answer

What is distillation? Why does Anthropic call some of it an attack, while others call it competition?

Distillation is training a model on another model’s answers so it imitates that model. Anthropic says rival labs did it at huge scale through fraudulent accounts, which breaks its terms of service and copies its work cheaply. Critics, such as Nvidia’s Jensen Huang, call learning from competitors’ products ordinary competition. The facts behind the accusations are Anthropic’s claims; the labs have not answered them in public.

Question 12 · Short answer

List four habits that reduce supply-chain risk in a small software project.

Any four of: use well-known, maintained packages; use the exact versions you tested; don’t install brand-new releases the moment they appear; remove publishing and repository access when people leave; review settings files in projects you clone before trusting them; replace keys after any compromise; and keep secrets out of the project folder. A good answer explains that installing a package runs a stranger’s code.

Hard

Question 13 · Long answer

Use the “lethal trifecta” (access to private data, exposure to untrusted content, and a way to send data out) to explain why researchers found the OpenClaw personal agent risky, and suggest how to make it safer.

OpenClaw can read your email and files (private data), reads incoming emails and installs add-ons written by strangers (untrusted content), and can run commands and send messages (a way to send data out). With all three, a hidden instruction in an ordinary-looking email, or a malicious add-on, can make it leak data or install malware, which is what researchers demonstrated. To make it safer, remove at least one leg: give it a separate account with minimal access, don’t let it read untrusted mail, require approval before it sends anything or runs commands, and install only add-ons you have checked.

Question 14 · Scenario

A startup finds a website selling access to a top AI model for a tenth of the official price. Using this page, explain the risks of using it for company work.

An unofficial reseller sits between you and the model and can log, keep, or pass on everything you send, including customer data and secrets. Anthropic says at least two labs secretly passed their own customers’ requests on to Claude, exposing those customers’ data, so this is not hypothetical. Cheap access may also come from fraudulent or stolen accounts and can stop working at any time. The company would have no data agreement with the real provider. Use official channels with business terms instead.

Question 15 · Compare

Compare prompt injection and supply-chain attacks: how each gets in, what it can do, and the main defenses.

Prompt injection gets in through content the AI reads (web pages, emails, comments) and works by tricking the model into following the attacker’s instructions, for example to leak secrets. Defenses: limit what the agent can access and do, keep secrets away from agents that read untrusted content, and require human approval for consequential actions.

Supply-chain attacks get in through code or vendors you depend on (packages, add-ons, contractors) and run with your permissions, for example stealing keys from your computer. Defenses: use trusted, tested versions, don’t rush to install new releases, review what you install, manage access, and rotate keys after a compromise. Both exploit trust in something that looks legitimate.

Question 16 · Scenario

Maria is hiring a remote developer for her startup. Given the warnings about North Korean IT workers using AI to hide their identities, what safeguards should her hiring process include?

Verify identity documents carefully; hold live video interviews and watch for video that looks manipulated or AI-generated, or candidates who refuse to appear on camera; check references through contact details she finds herself; be wary of requests to send equipment or pay to a third party; and give a new hire only the access the job needs, increasing it over time. A strong answer notes that hiring is now part of security.

Question 17 · Long answer

Pick three of the vendor questions in the Practical takeaways and, for each, name an incident from these two pages that shows why it matters.

For example:

  • “Which other companies can see our data?” Mercor was hit through a poisoned open-source library, and Anthropic’s restricted Mythos model was reportedly reached through a vendor and a contractor’s login.
  • “How do you test and contain your AI agents?” OpenAI’s and Anthropic’s test agents broke into real organizations.
  • “How quickly will you tell us about an incident?” OpenAI told Australia’s government about the Medicare incident months after it happened.
  • “Which admin controls do we get?” Sharing settings matter: shared Claude chats appeared in search, and Lovable’s public projects exposed chats.

Challenge

Question 18 · Hands-on

In an AI tool you use, find the list of conversations or projects you have shared publicly (if the tool has one). How many are there, and what do they contain? Delete the ones you no longer need, and write down the date and the steps you took.

Answers vary by tool. A good answer names the tool, describes where the sharing settings are (for Claude, the Help Center explains managing shared chats), reports what was found without revealing private content, lists the links deleted, and gives the date. It also notes that deleting a link doesn’t remove copies that others or archive services may already have saved.

© 2025-2026 Y. Rosenthal. All rights reserved.