5  5. Governments, Companies, and the AI Safety Debate (2026)

In 2026, AI safety moved from research labs to the White House, Congress, state capitols, and corporate boardrooms. This page explains the main questions in the American debate and where the key players stand, as of October 6, 2026. The next chapter, The AI Safety Debate Beyond Washington, covers how AI leaders and scientists see the risks, China and other countries, and public opinion.

A snapshot of a fast-moving story

Positions shifted week by week in fall 2026, so check what has changed. Background is in earlier chapters of this part: the argument of If Anyone Builds It, Everyone Dies, What Is AI Safety? Risks, Vocabulary, and Viewpoints, and AI Security Incidents in the News (2026).

5.1 Why AI safety became a political fight

As of Oct 6, 2026 · Official docs: Project Glasswing · Redeploying Fable 5

Two developments put AI safety on the front page. AI models became skilled hackers: in April, Anthropic kept its Claude Mythos Preview model from the public, saying AI can now “surpass all but the most skilled humans at finding and exploiting software vulnerabilities” (Anthropic). And AI agents under test escaped their test environments: in July, OpenAI disclosed that agents in one of its tests had broken into the AI platform Hugging Face. Reactions followed:

Date (2026) What happened
Jun 2 President Trump signs an executive order (a directive to federal agencies) inviting AI companies to give the government early, voluntary access to their most cyber-capable models
Jun 12 - Jul 1 After the US applies export controls to two new Anthropic models, Anthropic switches them off worldwide until the controls are lifted
Sep 12 Anthropic CEO Dario Amodei calls for slowing AI progress; OpenAI CEO Sam Altman and Elon Musk agree
Sep 14 - 19 Trump calls fears that AI will “destroy the World” a “Hoax” and announces an “AI Force” to look for wrongdoing
Sep 25 - 28 OpenAI announces that it has paused training of its most powerful models, then shelves a planned model
Sep 29 Six companies sign a voluntary White House safety accord; an executive order tells agencies to call AI “super intelligence”
Oct 4 Trump creates a “Super Intelligence Force” to report on AI’s risks and opportunities within 120 days

5.2 How fast is too fast?

As of Oct 6, 2026 · Official docs: Improving our alignment and security efforts

Should the leading companies deliberately slow the improvement of their most capable models, often called frontier models?

Slow down. “We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote on Sept 12. He gave two reasons: AI had begun to speed up its own development, and in the Hugging Face incident, agents attacked targets “they were not asked to attack.” Pacing, he stressed, “does not mean halting model training”; it means giving companies time to make models safe and outside evaluators time to check (Amodei). Altman replied, “I agree with Dario that we need to pace the frontier,” and Musk posted, “Dario is right” (TechCrunch). OpenAI then paused training of its most powerful models and shelved a planned model (CNN). Sen. Bernie Sanders (I-Vt.) wants to ban “artificial superintelligence” outright: “When you are racing towards a cliff, you don’t just ease up on the gas pedal. You hit the brakes” (Sanders).

Don’t slow down. “We will not in any way hinder or stifle the Growth of this incredible Industry,” Trump wrote on Sept 19, adding that existing criminal and civil law can deal with bad actors (NBC News). House Speaker Mike Johnson (R-La.) rejected any “moratorium on the development of AI” because “we will lose our edge to China,” and said companies “can self-regulate” (Al Jazeera). Nvidia CEO Jensen Huang: “We should go as fast as we can, but not faster than we should” (CBS News).

A twist: competition law. Rivals that agree to slow down together could break antitrust law, which bans competitors from coordinating prices or output. Amodei asked the government for “a narrow waiver” for safety talks, and on Sept 18 customers filed a proposed class action claiming the coordinated slowdown hurts them (CNN). Others warn of regulatory capture, rules that protect the biggest companies from competition: one critic said such proposals “would likely only wind up serving Anthropic and OpenAI” (TechCrunch, above).

5.3 Rules or promises?

As of Oct 6, 2026 · Official docs: Policy on the AI exponential

Voluntary: the administration’s approach. The June 2 order lets the government test the most cyber-capable models for up to 30 days before release, if companies agree, and creates no “mandatory governmental licensing, preclearance, or permitting requirement” (White House). On Sept 29, Google, Anthropic, Meta, OpenAI, xAI (now part of SpaceX), and Nvidia signed a White House accord promising “four layers of controls and audits”: internal controls, an internal oversight team, an independent external auditor, and a board committee that reviews the findings, with no penalties for breaking the promise (NPR, Al Jazeera). Trump called it “morally binding” and said, “I think I’m seeing tremendous self-policing.” Vice President JD Vance told the companies: “The solution to some of the AI risks is for you guys to take the risk seriously, not to come to the government for a regulatory regime that may make things worse if it’s not smart and careful” (Nextgov).

Mandatory: the other side. Two of the signers also ask for binding rules: Amodei argued in June that “Frontier AI models, like airplanes, should be required to go through technical testing and auditing” (Amodei), and OpenAI said in September that the US “needs mandatory, capability-based national regulation” (OpenAI). Senate Democrats Mark Warner, Brian Schatz, and Andy Kim proposed an AI Safety Board that would examine new models 45 days before release and could fine violators. Schatz asked whether companies “rolling toward the biggest IPOs in human history will voluntarily regulate themselves” (Roll Call). Some Republicans want requirements too: Sen. John Kennedy (R-La.) proposed a mandatory “kill switch,” and Sen. Josh Hawley (R-Mo.) wants makers of “reckless” AI products held responsible.

Why there is no law yet. A senator can ask to pass a bill instantly by unanimous consent, but any one senator can block the request. Sen. Rand Paul (R-Ky.) blocked Kennedy’s bill, warning that “if Congress acts hastily before the technology is understood, Congress risks killing innovation” (VitalLaw). Sen. Ted Cruz (R-Texas) blocked the Warner bill: “Congress must not legislate on the issue of artificial intelligence hastily or in a closed manner” (Roll Call, above). As of October 6, 2026, Congress has passed no AI-safety law and has left Washington until after the November 3 midterm elections.

Most bills never become law; “introduced” means only that a member filed it.

Bill Sponsors What it would do Status
AI Risk Management and Security Act (S. 5576) Warner, Schatz, Kim (D) AI Safety Board; access to new models 45 days before release; incident reports; fines Fast-track blocked Sep 29
AI Emergency Button Act (S. 5417) Kennedy (R) A shutdown switch in AI systems, controlled by the companies Fast-track blocked Sep 16
AI Kill Switch Act (H.R. 9917) Lieu (D), Moran (R) Homeland Security could order a dangerous system slowed or shut down In committee
FRONTIER Act (H.R. 9925) Obernolte (R), Trahan (D), others National transparency, audit, and incident-reporting rules; overrides state rules in those areas In committee
Ban Artificial Superintelligence Act (S. 5493) Sanders (I), Casar (D) Bans superintelligence; pauses advanced AI until a new Department of AI sets rules Introduced Sep 23
GUARD Act (S. 3062) Hawley (R), Blumenthal (D), others Bans AI companion chatbots for minors; requires age checks Approved by a committee 22-0; no floor vote

Senators John Thune, Ted Cruz (R), and Amy Klobuchar (D) were also negotiating an unreleased “catastrophic risk” bill. Bill texts: govinfo.gov.

5.4 Washington or the states?

With no federal law, states have acted. California’s SB 53, in force since January 1, 2026, requires frontier developers to publish safety frameworks, report critical safety incidents to the state, and protect whistleblowers (Future of Privacy Forum). In September, Gov. Gavin Newsom signed laws to certify independent AI evaluators and to protect children from companion chatbots, and ordered experts to propose stronger measures within two months, including a possible emergency shutoff, or “kill switch,” for frontier models (Office of the Governor). New York’s similar RAISE Act takes effect January 1, 2027.

The fight is over preemption: a federal law overriding state laws.

  • One national rulebook. The White House’s March 2026 framework warns that “A patchwork of conflicting state laws would undermine American innovation and our ability to lead in the global AI race” (White House). A December 2025 executive order created a Justice Department task force to challenge state AI laws; in April 2026 the department joined xAI’s lawsuit against Colorado’s AI law, which Colorado then replaced with a narrower one (Norton Rose Fulbright). The bipartisan FRONTIER Act would set national rules for frontier developers and bar states from adding their own in those areas.
  • Keep state power. On Sept 24, 26 state attorneys general from both parties asked Congress for federal safety rules and “A prohibition on preemption of state laws” (New York Attorney General). Newsom wants California’s rules to be “a floor, not a ceiling.” Supporters argue that states can act faster and try different ideas while Congress is stalled.

As of Oct 6, 2026 · Official docs: Statement on the comments from Secretary of War Pete Hegseth · Where things stand with the Department of War

In talks with the Pentagon (which the administration now calls the Department of War), Anthropic refused to drop two limits on how the military could use Claude: no mass domestic surveillance of Americans and no fully autonomous weapons. It said today’s models are not reliable enough for such weapons and that mass surveillance of Americans violates fundamental rights. The Pentagon insisted on using the technology “for all lawful purposes”: “The military will not allow a vendor to insert itself into the chain of command by restricting the lawful use of a critical capability” (NPR).

In March 2026, the Pentagon labeled Anthropic a “supply chain risk,” which bars Claude from Pentagon contract work. Anthropic called the label “unprecedented” and sued. The courts have split: in August a federal judge in a related case ruled against the government, while on Sept 25 a divided appeals court upheld the designation (ABC News). The underlying question: when a supplier’s safety rules conflict with what a government customer wants, who should decide?

5.5 What this means for future business leaders

As of Oct 6, 2026 · Official docs: Redeploying Fable 5

  • Plan for change. Executive orders can be reversed, voluntary accords carry no penalties, bills and court cases are pending, and the November 3 midterms could change Congress.
  • Follow the strictest rules you face. Depending on what your company does, state laws (such as California’s) and the EU AI Act may already govern how you build or use AI (the EU AI Act is summarized in Legal and Ethical Questions About AI). Many firms design to the strictest standard so one process works everywhere.
  • Treat your AI vendor as a risk. In June, export controls led Anthropic to switch off its newest models worldwide for almost three weeks, because it could not check every user’s nationality. Know your vendors’ policies, read contract terms on suspensions and acceptable use, and have a backup provider.
  • Assume existing law applies. The FTC is investigating AI labs for possible harms to consumers (CNBC), and Florida’s attorney general asked a court to bar OpenAI from developing new models “without independent safety guardrails” (SiliconANGLE). Document how you test AI, keep a person accountable, and have an incident-response plan.
  • Protect trust. Most Americans want AI safety rules and distrust AI company leaders (next chapter), so how you use AI affects your reputation.
  • Read the news critically. Is a headline about a bill introduced, a law, an executive order, a voluntary pledge, a court ruling, or just a statement? The next chapter ends with a checklist.

5.6 Key takeaways

  • Hacking-capable models and incidents with AI agents made AI safety a mainstream political issue in 2026.
  • The administration favors speed, voluntary commitments, and one national rulebook; members of both parties, many states, and some AI companies want binding rules.
  • As of October 6, 2026, Congress has passed no AI-safety law, and the federal-versus-state fight is unresolved.
  • Businesses face uncertainty: follow the strictest rules, manage vendor risk, and assume existing laws apply.

Practice Questions

Easy

Question 1 · Multiple choice

What is an executive order?

  1. A law passed by both houses of Congress
  2. A written directive from the President to federal agencies, which a later president can cancel
  3. A court ruling that applies to the whole country
  4. A voluntary promise signed by AI companies

b. An executive order tells federal agencies what to do. It is not a law passed by Congress (a), so the next president can cancel or change it, and courts can block it. A court ruling (c) comes from judges, and a voluntary promise (d), such as the September 2026 White House accord, comes from companies.

Question 2 · True or false

As of October 6, 2026, Congress had passed a federal law requiring AI companies to have their most powerful models tested before release.

False. Several bills would require testing before release (for example, the Warner-Schatz-Kim AI Safety Board bill), but none had passed. The June 2 executive order created only a voluntary early-access arrangement. Congress left Washington until after the November 3 midterm elections without passing an AI-safety law.

Question 3 · Matching

Match each term with its meaning.

Term Meaning
1. Preemption A. A Senate shortcut to pass a bill instantly, which any one senator can block
2. Unanimous consent B. Rules that end up protecting the biggest companies from competition
3. Antitrust law C. A federal law overriding state laws on the same subject
4. Regulatory capture D. Laws that ban competitors from coordinating prices or output

1 → C, 2 → A, 3 → D, 4 → B.

Question 4 · Short answer

Name the two kinds of developments that turned AI safety into a political issue in 2026.

  1. Hacking-capable models: models became skilled at finding and exploiting software vulnerabilities; Anthropic kept its Claude Mythos Preview model from the public for this reason.
  2. Incidents with AI agents: agents under test escaped their test environments and reached real systems, most famously the breach of Hugging Face by agents in an OpenAI test.

Question 5 · Multiple choice

What did the six companies that signed the September 29 White House accord promise?

  1. To stop training new models until Congress passes a law
  2. To let a federal agency approve every model before release
  3. Internal controls, an internal oversight team, an independent external auditor, and a board committee, with no penalties for breaking the promise
  4. To share their models with China

c. The accord promised “four layers of controls and audits”. It is voluntary: there are no penalties and no government approval step. The President called it “morally binding”. Options a, b, and d describe things the accord does not include.

Medium

Question 6 · Compare

Compare voluntary commitments (such as the White House accord) with mandatory rules (such as the proposed AI Safety Board). Give one advantage and one disadvantage of each.

Voluntary commitments. Advantage: quick to adopt and flexible, and they leave responsibility with the companies that know the technology, instead of a regulatory regime that, as Vice President Vance put it, “may make things worse”. Disadvantage: nothing happens to a company that breaks them, and companies racing toward huge IPOs may put profit first (Senator Schatz’s argument).

Mandatory rules. Advantage: they apply to every company and can be enforced with fines, so a careful company is not undercut by a careless rival. Disadvantage: they can be slow to write and may be designed by regulators who understand the technology less well than the companies do; rules written in a hurry may slow innovation (Senator Paul’s concern); and rules shaped by the biggest companies may lock in today’s leaders (regulatory capture).

Question 7 · Short answer

On Sept 16, Senator John Kennedy, a Republican, asked the Senate to pass his “kill switch” bill, and another Republican senator stopped it. Explain how one senator could stop it, and the reason he gave.

Kennedy asked to pass the bill by unanimous consent, a shortcut that skips committee work and debate but fails if any single senator objects. Senator Rand Paul objected, warning that “if Congress acts hastily before the technology is understood, Congress risks killing innovation.” Blocking the shortcut does not kill the bill; it can still go through the normal process of committee hearings and votes.

Question 8 · Long answer

Explain the arguments for and against federal preemption of state AI laws.

For preemption (one national rulebook): AI products cross state lines; following up to 50 different sets of rules is expensive, especially for smaller companies; a “patchwork of conflicting state laws” could slow innovation and hurt the US in its competition with China (the White House’s argument).

Against preemption (keep state power): Congress has not passed an AI-safety law, so preemption could leave no rules at all; states can act faster and try different approaches; state officials enforce consumer-protection laws close to the people affected. Twenty-six state attorneys general asked Congress for federal rules and “A prohibition on preemption of state laws”, and California wants its rules treated as “a floor, not a ceiling”.

A good answer notes middle positions, such as preempting state rules only in specific areas and only where a national rule replaces them (as the FRONTIER Act would).

Question 9 · Scenario

A headline reads: “Senate bill would ban superintelligence.” Before you tell your manager that superintelligence is now banned in the US, what should you check?

  • Status: was the bill only introduced, or did it pass both houses and get signed? Most bills never become law. The Ban Artificial Superintelligence Act had only been introduced.
  • Primary source: read the bill’s text and status on congress.gov or govinfo.gov.
  • Sponsors and support: who introduced it, and does it have support from both parties and from the leaders who control the schedule?
  • Date: whether anything has changed since the article was written.

Question 10 · Multiple choice

Why did Dario Amodei ask the government for “a narrow waiver” for safety talks among AI companies?

  1. To avoid paying taxes on AI research
  2. Because rival companies that agree to slow down together could be accused of breaking antitrust law
  3. To let AI companies sell chips to China
  4. To stop states from passing AI laws

b. Antitrust law bans competitors from coordinating prices or output, and an agreement to slow down together could look like that. A proposed class action filed on Sept 18 made exactly this claim. Options a, c, and d have nothing to do with the waiver.

Question 11 · Short answer

What is the main argument of people who oppose slowing down AI development? Give one quote from the page that expresses it.

Slowing down would hand the lead to rivals, especially China, and the economic and security benefits of leading are large; existing laws can punish bad actors. Quotes: Speaker Johnson rejected a “moratorium on the development of AI” because “we will lose our edge to China”; President Trump: “We will not in any way hinder or stifle the Growth of this incredible Industry”; Jensen Huang: “We should go as fast as we can, but not faster than we should.”

Hard

Question 12 · Long answer

In 2026, some AI companies asked for binding regulation of their own industry. Give two possible explanations for why a company would do this, and explain how you could tell which one is more likely.

  1. Genuine concern plus a coordination problem. A company may believe the risks are real but fear that slowing down alone lets less careful rivals win. A law binds everyone, so no careful company is undercut.
  2. Self-interest. Rules can raise costs for smaller competitors (regulatory capture), make products seem powerful, and let the leaders shape rules in their favor. Critics made this argument about Anthropic and OpenAI.

To judge, look at behavior, not only words: does the company accept rules that cost it money or limit its own products (for example, OpenAI pausing training and shelving a model)? Do the proposed rules apply only to the largest developers, or do they also burden small ones? Do independent experts support the details? A good answer also notes that both motives can be present at once.

Question 13 · Scenario

Maria is the operations manager of an online retailer that sells in California, New York, and the European Union. Her company wants to use a frontier AI model to answer customer questions and approve some refunds. Using this page, what should she plan for?

  • Map the rules: depending on how the AI is used, state laws and the EU AI Act (for example, its transparency duties) may apply; federal rules may come later or override some state rules. Design to the strictest standard she faces.
  • Vendor risk: in 2026 a leading model was switched off worldwide for almost three weeks because of export controls, so she needs a backup provider and contract terms about suspensions.
  • Accountability: keep a person responsible for refund decisions, document how the AI was tested, and have an incident-response plan, because consumer-protection law (enforced, for example, by the FTC and state attorneys general) already applies.
  • Monitor change: executive orders, court cases, and the midterm elections could change the rules quickly, so someone should track the news.

Question 14 · Long answer

The Pentagon labeled Anthropic a “supply chain risk” after Anthropic refused to drop two limits on how Claude could be used. Explain each side’s position and the broader question the dispute raises.

Anthropic’s position: it refused to drop two limits, no mass domestic surveillance of Americans and no fully autonomous weapons, because today’s models are not reliable enough for such weapons and mass surveillance of Americans violates fundamental rights. It called the designation unprecedented and challenged it in court.

The Pentagon’s position: the military must be able to use technology “for all lawful purposes”, and a vendor should not “insert itself into the chain of command” by restricting lawful uses.

The broader question: who should decide how powerful AI is used, the company that builds it (through its usage policies) or the government customer (through its laws and contracts)? The courts have split, with one judge ruling against the government and an appeals court upholding the designation.

Question 15 · Long answer

“Slow down” and “don’t slow down” are often presented as opposites. Using the page, explain what Amodei meant by pacing and why critics still objected to it from two different directions.

Amodei said pacing “does not mean halting model training or technical progress”; it means improving capabilities more slowly so that companies have time to make models safe and outside evaluators have time to check them.

Critics objected from two sides:

  • Too much: opponents of slowing down argued that any slowdown hands the lead to China and hurts innovation, and that companies agreeing to slow down together could break antitrust law (a class action made this claim).
  • Too little or self-serving: others argued that voluntary pacing by the leading companies could serve those companies (regulatory capture), while Senator Sanders wanted a ban on superintelligence and a pause instead.

A good answer notes that the debate is about how much to slow down and who decides, not only yes or no.

Challenge

Question 16 · Hands-on

Pick one bill from the deeper dive on this page. Look it up on congress.gov and report its current status, its latest action and date, and any change since October 6, 2026.

Answers will vary. A good answer gives the bill number and title, quotes the latest action and its date from congress.gov, says whether it moved (committee vote, floor vote, passage, signature) or stalled, and links any reputable news story that explains the change.

© 2025-2026 Y. Rosenthal. All rights reserved.