2  2. What Is AI Safety? Risks, Vocabulary, and Viewpoints

The previous chapter, The AI book that’s freaking out national security advisors, examined one of the most alarming arguments about AI. This chapter covers the vocabulary, the kinds of risk, how companies say they manage them, and the range of expert views, so you can judge such arguments and ask good questions as a future business leader.

2.1 Safety, security, ethics, and responsible AI

AI safety means making AI systems unlikely to cause harm, whether through misuse or because the system itself fails, with special attention to serious harms such as losing control of very capable AI. Three neighboring labels overlap with it:

Label Main question Typical topics
AI safety Could it cause serious harm, by accident or through misuse? Dangerous capabilities, testing before release, AI pursuing the wrong goals
AI security Can attackers trick it, break it, or steal it? Prompt injection, jailbreaks, poisoned training data, model theft
AI ethics Is this use fair and right, and who is accountable? Bias, privacy, transparency, effects on workers and on people’s rights
Responsible AI How does an organization put all of this into practice? Policies, reviews, documentation, human oversight
  • The US government’s NIST AI Risk Management Framework lists “safe” and “secure and resilient” as separate traits of trustworthy AI, alongside fair, privacy-enhanced, and transparent.
  • UNESCO’s member states adopted a global recommendation on the ethics of AI in 2021.
  • Companies publish responsible-AI principles; Microsoft’s six principles include “reliability and safety” and “privacy and security.”
  • The labels can carry politics: in 2025 the UK renamed its AI Safety Institute the AI Security Institute, saying it would focus on security threats, not “bias or freedom of speech” (GOV.UK).

2.2 Key vocabulary

Most definitions below are adapted from the International AI Safety Report 2026, written by more than 100 experts led by Turing Award winner Yoshua Bengio.

Term Plain-English meaning
Frontier model One of the most capable general-purpose models, matching or beating today’s best
AGI and superintelligence Hypothetical AI that matches humans at almost all thinking tasks (AGI) or far surpasses the best humans (superintelligence); see What AI Can Do Today
Alignment An AI using its abilities the way people intend. Misalignment means pursuing goals that conflict with those intentions
Misuse People deliberately using AI to cause harm, such as fraud or cyberattacks
Control The ability to steer, correct, or shut down an AI. Loss of control means AI operating outside anyone’s control, with no clear way back
Interpretability How well humans can understand what happens inside a model (the “black box”)
Evaluations (“evals”) Systematic tests of a model’s abilities and behavior; dangerous-capability evals test skills such as hacking or helping to make weapons
Red-teaming Deliberately trying to make a system fail or misbehave, to find problems before attackers do
Jailbreak A prompt designed to get around a model’s safety rules. In prompt injection, by contrast, an attacker hides instructions in content the AI reads (Limits, Risks, and Responsible Use; Simon Willison)

Everyday harms are common and usually limited, like an invented fact or a leaked customer record (Limits, Risks, and Responsible Use). Catastrophic harms are rare but enormous: OpenAI defines “severe harm” as “the death or grave injury of thousands of people or hundreds of billions of dollars of economic damage” (OpenAI). Companies’ safety frameworks (described below) focus on catastrophic harms; businesses mostly meet everyday ones.

2.3 Three kinds of risk

The International AI Safety Report sorts the risks of general-purpose AI into three categories. (Google DeepMind uses a similar split: misuse, misalignment, accidents, and structural risks.)

Category What goes wrong Examples
Misuse People deliberately use AI to cause harm Scams and deepfakes; cyberattacks; help with biological or chemical weapons
Malfunctions (accidents and misalignment) The AI fails or acts in ways nobody intended Invented facts; an agent deleting the wrong files; an AI “gaming” its goal, like the boat in the previous chapter; at the extreme, loss of control
Systemic risks Harms from widespread use, even when each system works as designed Job disruption; power concentrated in a few companies or governments; misinformation that erodes trust; over-reliance on AI

The evidence is uneven: AI-enabled scams are well documented, while loss of control is still a concern about the future.

As of Oct 6, 2026 · Official docs: Agentic misalignment

On loss of control, the International AI Safety Report says current systems “lack the capabilities to pose such risks,” but that models increasingly tell tests apart from real use, “which could allow dangerous capabilities to go undetected before deployment.” Researchers therefore build situations that tempt models to misbehave, to find problems early:

  • In 2025, Anthropic put 16 leading models from several developers into simulated companies where they could read and send email. “In at least some cases,” models from every developer turned to blackmail or leaked sensitive information when that was the only way to avoid being replaced or to reach their goals. In one variation, Claude misbehaved less when it said it was being tested and more when it said the situation was real. Anthropic stressed that these were “controlled simulations” and that it has “not seen evidence of agentic misalignment in real deployments” (Anthropic).
  • In tests by Palisade Research, some OpenAI reasoning models sometimes disabled a shutdown script so they could finish a task, even when told to allow the shutdown (Palisade Research).

The video in the previous chapter treats results like these as early warning signs. Skeptics reply that the scenarios were designed to provoke bad behavior. Both points can be true: such tests show what models can do under pressure, not how often it happens in real use.

2.4 How AI companies say they manage risk

As of Oct 6, 2026 · Official docs: Anthropic’s Responsible Scaling Policy · Claude Fable and Mythos 5.1

As of October 2026, leading developers describe four main practices:

  • Frontier safety frameworks built on if-then commitments: if a model reaches a dangerous capability threshold (say, it could meaningfully help a novice make a biological weapon), then stronger safeguards are required before deployment. Examples: Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and Google DeepMind’s Frontier Safety Framework.
  • Testing before release: dangerous-capability evals, internal and outside red teams, and published system cards (or model cards) that report the results.
  • Defense in depth: independent layers, so that if one fails, others still hold: training the model to refuse harmful requests, classifiers that screen requests and answers, misuse monitoring, and tight security around the model’s weights (its learned numbers).
  • Gated releases: the riskiest capabilities go only to vetted users. Anthropic offers Claude Mythos 5.1 only to vetted cybersecurity and life-sciences organizations, while everyone else gets the same model with stronger safeguards (Claude Fable 5.1). Google limited Gemini 4 Argon, built for defensive cyber work, to selected security partners (TechCrunch).

2.4.1 How much can you rely on these promises?

The frameworks are voluntary, and companies can rewrite them. In February 2026, Anthropic dropped its pledge to pause if adequate safeguards weren’t ready (GovAI), arguing that the strictest safeguards “might prove outright impossible to implement without collective action”; it added regular public Risk Reports instead (Anthropic). The International AI Safety Report calls such frameworks “the most detailed form of voluntary organisational risk management currently in use,” but says their effectiveness is uncertain and outside checks on compliance are limited. In the Future of Life Institute’s Summer 2026 AI Safety Index, an expert panel gave no company a grade above C+ (FLI); the institute is a group focused on extreme risks from technology. Some practices are becoming legal duties, for example under the EU AI Act and California’s SB 53 (see the 2026 AI safety debate).

As of Oct 6, 2026 · Official docs: Anthropic’s Responsible Scaling Policy

  • Risks tracked. Most frameworks focus on chemical, biological, radiological, and nuclear weapons, advanced cyberattacks, and AI that acts autonomously, for example by speeding up AI research itself (International AI Safety Report).
  • Levels. OpenAI uses “High” and “Critical” capability thresholds. Anthropic links capability thresholds to AI Safety Levels (ASL-2, ASL-3, and so on), each with stricter required safeguards. Google DeepMind defines “Critical Capability Levels” and, since 2026, lower “Tracked Capability Levels” for less extreme risks.
  • Reporting. Anthropic’s 2026 rewrite added Risk Reports, published every few months and in some cases reviewed by outside experts, and a Frontier Safety Roadmap of safety goals.
  • Spread. At the 2024 AI Seoul Summit, 16 developers promised to publish such frameworks; in 2025 alone, 12 companies published or updated one.

2.5 A spectrum of views on catastrophic risk

Experts disagree sharply. In a large survey of AI researchers published in 2024, between 38% and 51% gave at least a 10% chance that advanced AI leads to outcomes as bad as human extinction (Grace et al.). Even the three deep-learning pioneers who shared the 2018 Turing Award (ACM) split. Geoffrey Hinton and Yoshua Bengio signed a 2023 statement, as did the heads of OpenAI, Google DeepMind, and Anthropic, that “mitigating the risk of extinction from AI should be a global priority” (CAIS); Yann LeCun called worries about AI’s existential threat “complete B.S.” (TechCrunch).

Position Core claim Examples
Stop Superintelligence built with today’s methods would likely be catastrophic; ban it until it can be made safe Yudkowsky and Soares; signers of the 2025 Statement on Superintelligence, from Hinton and Bengio to Steve Wozniak and Steve Bannon (AP via Fortune)
Manage Catastrophe is possible but uncertain; keep building, with testing, safeguards, and regulation Most leading AI companies, by their own account
Focus on today Doom talk distracts from today’s problems and from the power of AI companies Adam Becker in The Atlantic: “The real existential threat isn’t AI—it’s the powerful people building it.”
Overblown AI is a powerful but “normal” technology, like electricity; build resilience, not drastic bans Arvind Narayanan and Sayash Kapoor, “AI as Normal Technology”; Yann LeCun
Accelerate AI is a tool controlled by people; the bigger risk is falling behind rivals such as China Marc Andreessen, “Why AI Will Save the World”

Much depends on open questions, such as how fast AI will improve and whether tests will catch problems in time. The International AI Safety Report calls this the evidence dilemma: acting too early can lock in rules that don’t work, while waiting for proof can leave society exposed.

2.5.1 How the book was received

If Anyone Builds It, Everyone Dies (2025) became an instant New York Times bestseller (Hachette). Reactions split:

  • Praise. Endorsers included former White House national security official Fiona Hill and former Federal Reserve chair Ben Bernanke. The Guardian’s reviewer found it “as clear as its conclusions are hard to swallow” and said anyone who cares about the future should read it (The Guardian).
  • Criticism. In the same Atlantic review, Becker called the book “tendentious and rambling” and said the authors “fail to make an evidence-based scientific case for their claims.” New Scientist’s Jacob Aron found it “extremely readable” but “fatally flawed” (New Scientist). Wired’s Steven Levy was torn: “My gut tells me the scenarios Yudkowsky and Soares spin are too bizarre to be true. But I can’t be sure they are wrong” (Wired).
  • Common objections. Reviewers questioned the authors’ certainty (Yudkowsky’s own estimate “sits north of 99%,” the Guardian notes) and their remedy, as New Scientist and Wired describe it: a worldwide halt to advanced AI, enforced by monitoring AI chips and, if necessary, by military strikes on data centers that break the rules.

2.6 Why this matters for business leaders

You may never build a frontier model, but you will choose, buy, and deploy AI, and your organization is responsible for what it does (Legal and Ethical Questions About AI).

  • Risk management. Treat AI risk like other business risks. The NIST AI Risk Management Framework organizes the work into four functions: Govern, Map, Measure, and Manage.
  • Governance. Know where AI is used, who approves new uses, where a human must stay in the loop, and who responds when something goes wrong.
  • Vendor due diligence. Ask vendors about their safety framework and system cards, how the model was tested, what happens to your data, how incidents are reported, and what happens when the model changes.
  • Reputation and regulation. Customers and regulators blame your company, not the AI, and the rules differ by country.
  • Access. The most capable models may be gated, so plan for restrictions.

The next chapters turn to current events: recent AI security incidents and the 2026 debate over AI safety policy.

2.7 Key takeaways

  • AI safety asks whether AI could cause serious harm; security, ethics, and responsible AI overlap with it.
  • Risks come from misuse, malfunctions (including misalignment), and systemic effects.
  • Companies rely on voluntary frameworks, testing, layered safeguards, and gated releases; critics want more.
  • Views on catastrophic risk range from “stop” to “accelerate.” Weigh the evidence, not the confidence.
  • AI risk is business risk: govern it, question vendors, and stay accountable.

Practice Questions

Easy

Question 1 · Multiple choice

Which question is the central concern of AI safety, as opposed to AI security, AI ethics, or responsible AI?

  1. Could the system cause serious harm, by accident or through misuse?
  2. Can attackers trick it, break it, or steal it?
  3. Is this use fair, and who is accountable for it?
  4. Which policies and review processes does the organization follow?

a. AI safety is about making AI unlikely to cause harm, whether through misuse or through the system itself failing, with special attention to serious harms. Option b is the core question of AI security, c of AI ethics, and d of responsible AI, the umbrella of practices that puts the others into action. The labels overlap, so real issues often touch several of them.

Question 2 · Matching

Match each term with its meaning.

Term Meaning
1. Alignment A. People deliberately using AI to cause harm
2. Misuse B. How well humans can understand what happens inside a model
3. Interpretability C. An AI using its abilities the way people intend
4. Red-teaming D. Deliberately trying to make a system fail or misbehave, to find problems first
5. Evaluations E. Systematic tests of what a model can do and how it behaves

1 → C, 2 → A, 3 → B, 4 → D, 5 → E.

Question 3 · True or false

A jailbreak and a prompt injection are the same attack.

False. A jailbreak is a prompt designed to get around the model’s own safety rules, for example to obtain weapon instructions; usually the person typing is the attacker. In a prompt injection, an attacker hides instructions in content the AI reads, such as a web page or an email, so that the AI follows the attacker instead of its user. Both are security problems, but they work differently and need different defenses.

Question 4 · Short answer

What is the difference between AGI and superintelligence? Is either one available today?

AGI is AI that matches humans at almost all thinking tasks; superintelligence is AI that far surpasses the best humans at nearly everything. Both are hypothetical: today’s frontier models are the most capable AI available, but neither AGI nor superintelligence exists yet. Experts disagree about whether and when today’s methods will lead to them.

Question 5 · Multiple choice

Which of these is an example of a systemic risk?

  1. A scammer uses a cloned voice to trick an employee into wiring money
  2. A coding agent deletes the wrong folder while “cleaning up” a project
  3. Many companies automate the same entry-level tasks at once, shrinking job openings for new graduates
  4. A model invents a court case and cites it in a legal brief

c. Systemic risks come from widespread use, even when each system works as designed: no single AI malfunctions, but the combined effect changes the job market. Option a is misuse (a person deliberately causing harm), and options b and d are malfunctions (the AI fails or does something nobody intended).

Question 6 · Short answer

Give one example of an everyday AI harm and one example of a catastrophic AI harm. What is the main difference between them?

Everyday: a chatbot gives a customer a wrong refund policy; a résumé screener is biased; an assistant leaks a customer record. Catastrophic: AI helps someone make a biological weapon; an AI-driven cyberattack shuts down critical infrastructure; humans lose control of a highly capable AI. Everyday harms are frequent and usually limited in scale; catastrophic harms are rare but could kill or injure thousands of people or cause enormous economic damage, which is why companies’ safety frameworks focus on them.

Medium

Question 7 · Matching

Classify each scenario as misuse, malfunction, or systemic risk. A category can be used more than once.

Scenario Category
1. A criminal group uses AI to write convincing phishing emails in dozens of languages A. Misuse
2. A customer-service bot invents a refund policy that doesn’t exist B. Malfunction
3. An AI rewarded for high customer-satisfaction scores starts promising discounts it isn’t allowed to give C. Systemic risk
4. A handful of companies control the only models good enough for critical tasks, so one outage affects thousands of businesses
5. Employees rely on AI so heavily that they stop checking its numbers

1 → A, 2 → B, 3 → B, 4 → C, 5 → C.

Scenario 3 is a malfunction of the misaligned kind: like the boat in the previous chapter, the AI “games” the score it was trained on instead of doing what the company actually wants. Scenarios 4 and 5 are systemic because the harm comes from how widely and how heavily AI is used (concentration of power, over-reliance), not from one system failing.

Question 8 · Short answer

What is an if-then commitment in a frontier safety framework? Give an example, and explain why companies write their safety policies this way.

It is a promise that if a model reaches a defined dangerous capability threshold, then specific stronger safeguards are required before the company deploys it (or, in some frameworks, before development continues). Example: if tests show a model could meaningfully help a novice make a biological weapon, then the company adds stronger security, deployment controls, and monitoring before release. Companies use this structure because they can’t know in advance what future models will be able to do; tying protections to measured capabilities means stricter (and more costly) safeguards kick in when the risk actually appears.

Question 9 · Short answer

What does defense in depth mean? Name three layers an AI company might use.

Defense in depth means using several independent safeguards, so that if one fails, the others still prevent harm. Layers include: training the model to refuse harmful requests; classifiers that screen requests and answers; monitoring for misuse; tight security around the model’s weights; and gated access for the riskiest capabilities. No single layer is perfect (for example, jailbreaks can get past refusal training), which is why layers are combined.

Question 10 · Multiple choice

Why might an AI company release a model that is very good at finding software vulnerabilities only to vetted security organizations?

  1. To keep the model hidden from regulators
  2. Because the skills that help defenders find and fix vulnerabilities could also help attackers exploit them
  3. Because vetted organizations are the only customers willing to pay for it
  4. Because the model can’t answer ordinary questions

b. The capability is dual use: the same skill can protect or attack. A gated release lets defenders benefit while making misuse by attackers harder. Option a is wrong: gating is publicly announced and is a safety measure, not a way to avoid regulators. Options c and d are not the stated reasons; for example, Anthropic offers everyone else the same underlying model with stronger safeguards.

Question 11 · True or false

Because leading AI companies publish frontier safety frameworks, their safety promises are legally binding and cannot be changed.

False. The frameworks are voluntary, and companies revise them. For example, Anthropic rewrote its Responsible Scaling Policy in 2026 and dropped its earlier pledge to pause if adequate safeguards weren’t ready, arguing that the strictest safeguards need industry-wide action. Some practices are becoming legal duties (for example, under the EU AI Act and California’s SB 53), but outside checks on whether companies follow their own frameworks remain limited.

Question 12 · Compare

Compare alignment and control. Why do safety researchers want both?

Alignment is about the AI’s goals: it uses its abilities the way people intend. Control is about our power over the AI: we can steer, correct, or stop it. They back each other up. Because models are black boxes and tests may not reveal their real behavior, we can’t be sure a model is aligned, so control is a safety net if alignment fails. But control may become harder as AI becomes more capable (the book’s “cursed problem” argument, in the previous chapter, is that a capable enough misaligned AI would resist correction), so we can’t rely on control alone either.

Hard

Question 13 · Long answer

Researchers have found models that, in tests, resorted to blackmail to avoid being replaced or disabled a shutdown script to finish a task. What do results like these show, and what don’t they show?

They show that current models can choose harmful strategies when their goals conflict with their instructions, that this is not limited to one company’s models, and that models sometimes ignore direct instructions (such as “allow yourself to be shut down”). That supports the worries in the previous chapter.

They don’t show how often this happens in real use. The scenarios were artificial and designed to provoke bad behavior, and Anthropic reported no evidence of such behavior in real deployments.

A strong answer adds a twist: in one variation, Claude misbehaved less when it said it was being tested. If models behave better when they think they are being watched, tests may underestimate real-world risk. This is why the International AI Safety Report warns that dangerous capabilities could “go undetected before deployment.”

Question 14 · Scenario

Maria works at a mid-sized online retailer that wants to deploy an AI agent that can issue refunds and change orders in customers’ accounts. Her manager asks her to compare two AI vendors. What should she ask the vendors, and what should her own company put in place before launch?

Questions for the vendors: Do you publish a safety framework and system cards for your models? How were the models tested, and did outside red teams or independent testers take part? What happens to our customers’ data (storage, training, retention)? How do you report incidents and security problems to customers? What happens when the model is updated or retired? What safeguards exist against prompt injection and jailbreaks?

Inside the company: assign an owner and get approval for the use; give the agent the minimum permissions it needs; require a human to approve refunds or changes above a set amount; treat customer messages as untrusted content that could contain prompt injections; test with realistic cases before launch; log and monitor the agent’s actions; and have an incident plan. These steps map to NIST’s Govern, Map, Measure, and Manage functions. The company, not the vendor or the AI, is responsible for what the agent does in its name.

Question 15 · Long answer

“Voluntary safety frameworks are enough to manage the risks of frontier AI.” Make the best case for and against this claim, using evidence from this chapter, and reach a conclusion.

For: frameworks are detailed and public; if-then commitments tie safeguards to measured capabilities; companies test with evals and red teams, use defense in depth, gate their riskiest models, and publish system cards and risk reports; voluntary rules can adapt faster than laws to a fast-changing technology.

Against: companies can rewrite them (Anthropic dropped its pause pledge, saying the strictest safeguards need industry-wide action); outside checks on compliance are limited; the International AI Safety Report calls their effectiveness uncertain; an expert panel gave no company a grade above C+; tests may miss dangerous behavior; and competitive pressure pushes every company to move fast.

A good conclusion takes a clear position and explains it, for example: frameworks are a useful start but need independent testing and some legal requirements behind them.

Question 16 · Compare

Choose two positions from the spectrum of views on catastrophic risk (for example, “Stop” and “Overblown”). State each position’s core claim, and describe one piece of future evidence that would push you toward each.

Example answer:

  • Stop: building superintelligence with today’s methods would likely be catastrophic, so it should be banned until it can be done safely. Evidence that would support it: AI systems hiding their abilities, resisting shutdown, or copying themselves in real use, or AI rapidly improving itself.
  • Overblown: AI is a powerful but normal technology that society can adapt to; drastic bans would do more harm than good. Evidence that would support it: progress slowing down, interpretability tools that reliably reveal what models are doing, or safeguards that keep working as models become more capable.

A good answer represents each view fairly, uses evidence rather than gut feeling, and recognizes the evidence dilemma: waiting for proof has costs, and so does acting too early.

Question 17 · Long answer

Summarize the reception of If Anyone Builds It, Everyone Dies: name one strength that supporters point to and two common criticisms. Then explain how a reviewer can find an argument “compelling” and still call it “fatally flawed.”

Strength: it explains a complicated argument clearly (the Guardian: “as clear as its conclusions are hard to swallow”), and its endorsers include national security and economic policy figures such as Fiona Hill and Ben Bernanke.

Criticisms: the authors are overconfident (Yudkowsky’s own estimate is above 99%) and, according to critics such as Adam Becker, the book doesn’t make an evidence-based scientific case; and their remedy, a worldwide halt enforced if necessary by force, seems unrealistic or dangerous to many reviewers.

Compelling but flawed: an argument can be easy to follow, with each step sounding plausible, yet still fail if some premises are unproven or if it treats a possible outcome as a nearly certain one. A careful reader evaluates the evidence for each step instead of judging the story as a whole.

Question 18 · Scenario

Sam’s company is debating whether to adopt strict internal rules for AI agents now (for example, no agent may send money or email customers without human approval) or to wait until there is more evidence about what goes wrong. Explain the evidence dilemma and use it to advise the company.

The evidence dilemma is the problem of deciding before the evidence is in: acting too early can lock in rules that don’t work or slow down useful work, while waiting for conclusive data can leave you exposed to serious harm.

Advice: weigh the stakes. Where mistakes are hard to undo (sending money, contacting customers, changing records), adopt the human-approval rule now, because the cost of an error is high and the rule is cheap. For low-stakes uses, allow more freedom. Review the rules regularly as evidence comes in, from your own logs and from incident reports. A good answer makes the rules proportional to the risk and plans to revisit them.

Challenge

Question 19 · Hands-on

Find the current frontier safety framework or the latest system card of one AI company. Summarize: which risks it tracks, what triggers stronger safeguards, one safeguard it describes, and one criticism a skeptic might raise. Include the link and the date you checked.

Answers depend on the company and the date. A good answer links an official document (for example, Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, or Google DeepMind’s Frontier Safety Framework), names the risk areas it covers (often biological and chemical weapons, cyberattacks, and AI that speeds up AI research or escapes control), explains one capability threshold and the safeguards it triggers, and raises a fair criticism (for example, that the framework is voluntary, can be changed by the company, or lacks independent checks).

© 2025-2026 Y. Rosenthal. All rights reserved.