6  6. The AI Safety Debate Beyond Washington (2026)

The previous chapter followed the fight over AI rules in Washington and the state capitals. This page widens the view: what the people building and studying AI say about the risks, how the US and China handle the race between them, the push for international rules, and what the public thinks, as of October 6, 2026. It ends with a checklist for reading AI-safety news critically. (For the broader spectrum of views, see What Is AI Safety? Risks, Vocabulary, and Viewpoints.)

6.1 What the people building AI say about the risks

As of Oct 6, 2026 · Official docs: Policy on the AI exponential · Improving our alignment and security efforts

Leaders of AI companies agree that their technology carries risks, but they disagree about how serious the risks are and what to do about them.

Leader On the risks On the rules
Dario Amodei, CEO of Anthropic “If managed poorly, I even believe that AI could be a risk to humanity as a whole” (UN, Sep 23) Binding testing and audits, as for airplanes; slow the pace of progress
Sam Altman, CEO of OpenAI “We could lose control of the future to AI” (UN, Sep 23), yet the world “should accept some bad things happening” for AI’s benefits (Oct) OpenAI supports “mandatory, capability-based national regulation”
Demis Hassabis, co-founder of Google DeepMind Calls for “proceeding with cautious optimism” (Jul) An independent standards body that reviews models before release: voluntary at first, required later
Elon Musk, CEO of SpaceX, which owns xAI Backed Amodei’s call to slow down: “Dario is right” (Sep 12) Regulation “does tend to be very much a one-way ratchet”; companies should test each other’s models before release (Sep 15)
Mark Zuckerberg, CEO of Meta “labs have a strong natural incentive to make their models more aligned” (Sep 15) No industry-wide slowdown; market forces will push companies to be safe
Mustafa Suleyman, CEO of Microsoft AI “We shouldn’t be trying to design models that can recursively self-improve beyond our control” (Sep 14) “Now’s the time for coordination” among labs
Jensen Huang, CEO of Nvidia Doom warnings are “too much drama,” though “Their concern is not wrong” (Sep 20) “Safety is an engineering problem, not a legal one” (Sep 15)

Sources: UN transcript, OpenAI, Fortune, Hassabis, TechCrunch, CNBC, Fortune, Fortune, CBS News, TechCrunch.

Why would CEOs warn about their own products? One reading is that the fear is sincere: “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt,” wrote Jacob Coxon, a researcher who quit Anthropic in September (TechCrunch). In July, more than 1,000 AI company employees, including Amodei and OpenAI’s chief scientist, asked the US government to support an international effort to “deliberately pace the frontier” (Pacing the Frontier). The other reading is that dramatic warnings make products look powerful, and rules shaped by the leaders can shut out smaller rivals (regulatory capture). Gary Marcus, a scientist who supports strong regulation, saw “self-congratulatory hype” in Altman’s and Amodei’s UN speeches (Marcus). Both can be true, so judge companies by what they do, such as pausing training or admitting outside evaluators, as well as by what they say.

6.2 Scientists and other voices

  • Yoshua Bengio, a Turing Award winner who chairs the International AI Safety Report (written by more than 100 experts and backed by more than 30 countries and international organizations), told the UN Security Council: “The dangers are real and imminent.” He wants frontier AI “licensed like other critical technologies,” as in medicine, aviation, and nuclear energy.
  • Geoffrey Hinton, a Nobel laureate, told reporters after a closed-door briefing for lawmakers in September that Congress may have only about one year left to impose safety measures (Fortune).
  • Focus on today’s harms. The Electronic Frontier Foundation (EFF), a digital-rights group, argued that the most urgent problems “are not about sci-fi scenarios concerning rogue super-intelligence” but “biased algorithmic decision-making,” “AI-powered surveillance,” and “artificially inflated personalized pricing” (EFF).

On Sept 15, progressives such as Sen. Bernie Sanders (I-Vt.) and conservatives such as former White House strategist Steve Bannon and Rep. Chip Roy (R-Texas) appeared together at a “Pro-Human Assembly” in Washington to raise alarms about AI’s risks. Sanders: “A super intelligent AI that escapes human control will not simply be an American problem. It will not be a Chinese problem. It will be humanity’s problem.” Roy: “I don’t think big tech CEOs should be making all the decisions about AI. I don’t think Congress should be making all the decisions about AI” (CNBC).

Religious leaders have joined the debate too. Pope Leo XIV’s encyclical on AI, Magnifica Humanitas (May 2026), calls for “adequate regulatory tools capable of upholding justice and curbing the distorting effects of technological power” (Vatican).

6.3 The United States and China

The race argument says that whoever leads in AI gains economic and military power, so slowing down could hand the lead to China. After meeting Chinese President Xi Jinping, Trump said the US is “not going to be putting on brakes” because “we’re leading China by a lot” (PBS). Jay Clayton, the Director of National Intelligence chosen to lead the new Super Intelligence Force, said: “The risk of not being first is high” (ABC News). Critics answer that the race is a choice: “The race is not a law of nature. It is the product of choices, choices made by the companies themselves,” Bengio told the UN. Microsoft CEO Satya Nadella pointed out that China faces the same dangers: “It’s not like they won’t have the same hacking problem” (TechCrunch, above).

The rivals also talk. During Xi’s state visit in late September, the two governments agreed to start a dialogue on AI’s risks and benefits, with the next meeting by November 2026, and to set up a “bilateral communication channel” for AI incidents, much like a hotline (White House, China’s Foreign Ministry). Treasury Secretary Scott Bessent said that “moving from opaque to more transparency between the number one and the number two AI powers in the world is very important” (NBC News). Meanwhile, the US limits sales of advanced AI chips to China with export controls (government limits on selling sensitive technology abroad), although in December 2025 it let Nvidia sell its H200 chips to approved customers in China, with 25% of the sales going to the US government (CNBC).

6.4 The push for global rules

  • On Sept 21-22, leaders of 20 countries, including Germany, Canada, Australia, Spain, and Singapore, and the European Commission issued “A Call for Control of Frontier AI Models.” It asks for “mandatory pre-deployment testing and independent evaluation” and for UN members to “explore creating an international institution, able to set standards, enable verification, and convene states when capability thresholds are crossed” (Government of the Netherlands). The US, China, the UK, France, India, and Japan did not sign (Al Jazeera).
  • On Sept 22, Trump told the UN General Assembly that the US “totally rejects any attempt to construct a globalist scheme of control for the Artificial Intelligence being spoken of so much now” (White House).
  • On Sept 23, the UN Security Council held a meeting focused on the safety risks of advanced AI (see the deeper dive below).
  • The UN’s new Independent International Scientific Panel on AI warned in its first brief, on the Hugging Face incident, that “the traditional model of safeguarding is unravelling” (UN News).
  • Since August 2, 2026, the EU’s AI Office can enforce the AI Act’s rules for general-purpose AI models and fine companies that break them (European Commission). Companies from anywhere that offer such models in the EU must comply (the EU timeline is in Legal and Ethical Questions About AI).

France convened the meeting. Highlights from the UN transcript (an automatic transcript, so some wording may be slightly off):

  • Yoshua Bengio: “The dangers are real and imminent.”
  • Sam Altman, OpenAI: “This moment calls for extreme care.”
  • The United States (Michael Kratsios, director of the White House Office of Science and Technology Policy): the rapid advance of AI “is not a reason to pause its further development or to constrain it with new global governance structures” (US Mission to the UN).
  • The United Kingdom (Foreign Secretary): “we cannot outsource to private companies the first duty of government to protect our people.”
  • China: asked countries to “support the UN as the main forum of global AI governance” and promoted its new Shanghai-based World AI Cooperation Organization.
  • Russia: “it is counterproductive to consider the use of AI technologies solely through the lens of risk.”

Other AI company leaders also spoke; their main warnings are in the table at the top of this page.

6.5 What the public thinks

Polls taken during the September debate show broad support for safety rules:

  • In a Quinnipiac poll of US adults (Sept 24-27), 86% supported requiring AI companies “to meet independent safety standards, even if it slows down AI development.” Asked about the pace of development, 30% said stop until safety can be evaluated, 47% slow down, 14% keep the current pace, and 5% move faster. And 74% had little or no trust in the leaders of AI companies (Quinnipiac).
  • The same poll found that 69% think keeping up with China is important and 91% think guardrails are important; 65% hold both views. As the pollster put it, “The public is not neatly divided into a competition camp and a safety camp.”
  • Worry differs by party. In an Economist/YouGov poll (Sept 18-21), 42% called AI’s risks “very serious,” up from 35% in late August, including 53% of Democrats and 29% of Republicans (YouGov). Among registered voters, 71% said they were more likely to back a candidate who supports stricter AI guardrails, including 52% of Republicans and 86% of Democrats (Quinnipiac).

Wording matters: a question that mentions slowing down, or one that mentions China, can shift the answers.

6.6 Reading AI-safety news critically

For a business leader, reading this news well shows which rules are coming, which are only talk, and what could disrupt your AI tools. A checklist:

  1. What kind of action is it? A bill introduced, a law, an executive order, a voluntary pledge, a court ruling, or just a statement? Only some of these bind anyone, and most bills never pass.
  2. Read the primary source. The Sept 29 executive order tells agencies to say “Super Intelligence” instead of “Artificial Intelligence,” but it defines super intelligence as the same technologies the law already calls artificial intelligence (White House). It renames today’s AI; it does not mean AI far smarter than humans, the sense used in the chapter on If Anyone Builds It, Everyone Dies.
  3. Compare official accounts. The White House called the new US-China channel the “U.S.-China Super Intelligence (SI) Dialogue”; China’s Foreign Ministry called it the “China-U.S. AI Dialogue” and did not mention the new term.
  4. Ask who is speaking and what they gain. CEOs, politicians, and advocacy groups all have interests. That doesn’t make them wrong, but it is a reason to check their claims.
  5. Check how a poll asked its question, and who was asked.
  6. Separate facts from forecasts, and check dates. “OpenAI paused training in September 2026” is a fact; “AI could kill us all by the end of the decade” is a forecast.
  7. Confirm with two reputable sources, and prefer original documents over social-media posts.

6.7 Key takeaways

  • AI leaders call the risks serious but split on remedies: binding rules and a slower pace, or market incentives and engineering fixes.
  • Scientists such as Bengio and Hinton urge fast government action; others stress today’s harms.
  • The US and China are racing, yet agreed to share AI-incident alerts. Many countries want international rules; the US government rejects global AI governance.
  • In fall 2026 polls, most Americans favored safety rules, though worry differed by party.

Practice Questions

Easy

Question 1 · Multiple choice

What does the White House’s September 29, 2026 executive order on “super intelligence” do?

  1. It announces that AI smarter than all humans now exists
  2. It tells federal agencies to call AI “super intelligence,” defining the term as the same technologies the law already calls artificial intelligence
  3. It bans the development of superintelligence
  4. It creates an international agency to regulate AI

b. The order is about terminology: agencies must say “Super Intelligence” instead of “Artificial Intelligence,” and the order defines super intelligence as the technologies already covered by the legal definition of artificial intelligence. It does not claim that AI smarter than humans exists (a), it bans nothing (c), and the US government rejects new global AI governance (d).

Question 2 · True or false

The United States and China both signed the September 2026 “Call for Control of Frontier AI Models.”

False. Twenty countries, including Germany, Canada, Australia, Spain, and Singapore, and the European Commission signed it. The US, China, the UK, France, India, and Japan did not. The US government said at the UN that it rejects a “globalist scheme of control” for AI.

Question 3 · Matching

Match each voice with the position it took in September 2026.

Voice Position
1. A Turing Award-winning AI scientist (Bengio) A. Today’s urgent problems are biased decisions, surveillance, and personalized pricing, not “sci-fi scenarios”
2. The CEO of a leading AI chip maker (Huang) B. Rapid AI progress is no reason to pause or to create new global governance structures
3. A digital-rights group (EFF) C. Frontier AI should be licensed like medicine, aviation, and nuclear energy
4. The US government at the UN Security Council D. “Safety is an engineering problem, not a legal one”

1 → C, 2 → D, 3 → A, 4 → B.

Question 4 · Short answer

What did the US and China agree to during Chinese President Xi Jinping’s state visit in September 2026?

To start a dialogue on the risks and benefits of AI, with the next meeting by November 2026, and to set up a bilateral communication channel for AI incidents, something like a hotline that lets each side alert the other.

Question 5 · Short answer

What are export controls? Give the AI-related example from this page.

Export controls are government limits on selling sensitive technology to other countries. The US limits sales of advanced AI chips to China; in December 2025 it allowed Nvidia to sell its H200 chips to approved customers in China, with 25% of the sales going to the US government. (The previous chapter describes export controls that briefly applied to Anthropic’s models in 2026.)

Medium

Question 6 · Long answer

Explain the race argument about AI and China, and give two counterarguments from this page.

The race argument: whoever leads in AI gains economic and military power, so slowing down could hand the lead to China. President Trump said the US is “not going to be putting on brakes,” and Jay Clayton said, “The risk of not being first is high.”

Counterarguments:

  • The race is a choice, not a law of nature (Bengio): companies and governments can decide to coordinate.
  • China faces the same dangers, such as hacking, so both sides have reasons to cooperate on safety (Nadella).
  • The two governments did agree to talk and to share AI-incident alerts, which shows rivals can cooperate on some risks.

Question 7 · Compare

Compare the view that AI may pose catastrophic risks (for example, Bengio’s) with the EFF’s view. Does accepting one view require rejecting the other?

Bengio warns that the dangers of advanced AI are “real and imminent” and wants frontier AI licensed like medicine, aviation, and nuclear energy. The EFF says the most urgent problems are already happening: biased automated decisions, surveillance, and personalized pricing.

They are not mutually exclusive. One can worry about future catastrophic risks and today’s harms at the same time. The real disagreement is about priorities: where attention, money, and new laws should go first. A good answer also notes that some measures, such as independent testing and incident reporting, address both concerns.

Question 8 · Scenario

A colleague says: “Polls show 86% of Americans want AI development stopped.” Is that what the Quinnipiac poll found? Correct the claim.

No. The 86% figure is the share who support requiring AI companies “to meet independent safety standards, even if it slows down AI development.” On a separate question about pace, 30% said stop until safety can be evaluated, 47% slow down, 14% keep the current pace, and 5% move faster. So most respondents favored stopping or slowing until safety can be evaluated, but only 30% chose “stop.” The lesson: quote the exact question and number, with the poll’s date.

Question 9 · Short answer

The White House and China’s Foreign Ministry gave the same US-China agreement different names. What were the names, and what does this teach you about reading the news?

The White House called it the “U.S.-China Super Intelligence (SI) Dialogue”; China called it the “China-U.S. AI Dialogue” and did not mention the new term. Each government describes events in the way that suits it. Compare official accounts from both sides, and other sources, before concluding what was agreed.

Question 10 · Multiple choice

Which statement is a forecast rather than a fact?

  1. OpenAI paused training of its most powerful models in September 2026.
  2. Twenty countries and the European Commission signed a call for an international AI institution.
  3. AI could kill us all by the end of the decade.
  4. Since August 2, 2026, the EU’s AI Office can fine makers of general-purpose AI models that break the AI Act.

c. It is a prediction about the future that can’t be checked today, however sincerely it is held. Options a, b, and d describe events or rules that can be checked against primary sources.

Question 11 · Long answer

Why would the CEO of an AI company warn that its own technology could be dangerous? Give two readings, and explain how you could judge between them.

  1. Sincere concern. The people closest to the technology may see risks first. A researcher who quit Anthropic wrote that the fear “is not a marketing stunt,” and more than 1,000 AI company employees signed a call to pace the frontier.
  2. Self-interest. Warnings make products seem powerful, and rules shaped by the leaders can shut out smaller rivals (regulatory capture). Critics such as Gary Marcus saw “hype” in some CEO speeches.

To judge, look at actions as well as words: does the company accept costs or limits, such as pausing training, withholding a model, or letting outside evaluators in? Do the rules it favors also bind itself? Both motives can exist at the same time.

Hard

Question 12 · Long answer

Should the US join an international institution that sets standards for frontier AI? Without taking a side, explain the strongest arguments on each side, using the positions on this page.

For joining: AI risks cross borders, and no country can manage them alone; shared testing standards, verification, and incident reporting reduce the chance of a race to the bottom (the 20-country Call for Control; Bengio; the UK’s view that governments cannot outsource safety to companies). The US-China incident channel shows that rivals can cooperate.

Against joining: a global regulator could limit national sovereignty, slow American innovation, or be shaped by rivals; national governments and responsible companies can manage risks themselves (the US government’s position at the UN; Trump’s rejection of a “globalist scheme”). Verification is hard, and rivals might not comply.

A strong answer weighs both sides and notes middle options, such as narrow agreements on specific risks or bilateral channels like the one agreed with China.

Question 13 · Scenario

Sam works for a US software company that sells an AI model to businesses in the US, the EU, and Asia. List the international risks from this page that his company should plan for.

  • EU rules: since August 2, 2026, the EU’s AI Office can enforce the AI Act’s rules for general-purpose AI models, with fines, so the company must comply in the EU.
  • More rules to come: 20 countries and the European Commission are pushing for an international institution and mandatory pre-deployment testing.
  • Geopolitics and export controls: US-China tensions can change which chips and models can be sold where, sometimes suddenly.
  • Public trust: polls show low trust in AI company leaders and strong support for safety standards, so customers may demand evidence of testing and oversight.

A good answer also says how to respond: track regulations by market, document testing, and plan for sudden changes.

Question 14 · Long answer

In the Quinnipiac poll, 65% of Americans said both that keeping up with China is important and that AI guardrails are important. What does this suggest for politicians and companies?

The public does not see a simple choice between competing with China and being safe; most people want both. For politicians, a message that frames safety and competitiveness as opposites may not match voters’ views; policies that pursue both (for example, testing requirements that don’t stop development) may have broad support. For companies, being seen as safe and trustworthy is part of competing, not a cost that can be skipped. A good answer adds a caution: poll results depend on wording, and opinions differed by party.

Question 15 · Scenario

A headline reads: “White House declares the era of superintelligence has begun.” Use the checklist on this page to evaluate it.

  • Kind of action: an executive order, which directs federal agencies; it is not a law, and a later president can change it.
  • Primary source: the order renames AI as “super intelligence” for federal agencies and defines the term as the same technologies the law already calls artificial intelligence. It does not say that AI smarter than humans exists.
  • Who is speaking: the administration, which says the new term better reflects the technology’s “transformative capabilities.” That is its stated reason, not a scientific finding.
  • Compare accounts: China’s readout of the same week’s agreement kept the term “AI”.
  • Conclusion: the headline is misleading if readers take “superintelligence” in its usual meaning, AI far smarter than humans.

Challenge

Question 16 · Hands-on

Find the latest news on one story from this page, such as the US-China AI dialogue (its next meeting was due by November 2026) or the UN discussions on an international AI institution. Compare an official source with two news reports.

Answers will vary. A good answer links the official source (for example, a White House or Chinese Foreign Ministry statement, or a UN page) and two reputable news reports, notes any differences in how they describe the event, separates facts from forecasts, and gives the date of each source.

© 2025-2026 Y. Rosenthal. All rights reserved.