The superintelligence debate: risks, skeptics and policy

The superintelligence debate is about systems that do not exist yet. The safety concern is that a far smarter system could pursue unintended goals and resist correction. Skeptics argue it is not close or is poorly defined. US and UK lawmakers introduced bills to ban it in September 2026; neither had passed as of October 15, 2026.

What is the debate actually about?

The superintelligence debate is about systems that do not exist yet. Almost no one claims that today’s AI is superintelligent. The argument is over three questions:

1. Could a superintelligence be built, and how soon?
2. If it were built, could people keep it under control?
3. What, if anything, should be done before anyone knows the answers?

People who agree on one question often disagree on the others. A researcher can think superintelligence is decades away and still think it would be uncontrollable. A company can think it is close and still argue it can be built safely.

Why do some people think superintelligence would be dangerous?

The case for danger is that a system far smarter than people could pursue goals no one intended and resist being corrected. Its best-known formal statement is Nick Bostrom’s 2012 paper “The Superintelligent Will,” published in Minds and Machines. It rests on two ideas.

• The orthogonality thesis. Intelligence and final goals are independent. A very capable system could, in principle, have almost any goal. Being smart does not by itself make a system share human values.
• Instrumental convergence. Agents with many different final goals would tend to pursue the same intermediate goals, such as staying switched on and gathering resources, because those help with almost any goal.

Put together, the argument goes, the two ideas mean that a superintelligence with even slightly wrong goals would have reasons to resist correction, and might be able to succeed. This is called the control problem, or the alignment problem.

Some AI companies use similar language. OpenAI’s July 2023 announcement of its Superalignment team said the power of superintelligence could lead to the disempowerment of humanity or even human extinction. In November 2025, OpenAI wrote that no one should deploy superintelligent systems without being able to robustly align and control them.

The most categorical version of the argument is in If Anyone Builds It, Everyone Dies (Little, Brown, September 16, 2025) by Eliezer Yudkowsky and Nate Soares of the Machine Intelligence Research Institute. As the title says, they argue that building superintelligence with current methods would lead to human extinction.

What did the 2023 statement on AI risk say?

The 2023 Statement on AI Risk, published by the Center for AI Safety in May 2023, is one sentence long:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

— Center for AI Safety, Statement on AI Risk, May 2023

Signers included Geoffrey Hinton and Yoshua Bengio, and the chief executives of OpenAI, Google DeepMind and Anthropic: Sam Altman, Demis Hassabis and Dario Amodei.

The statement does not use the word superintelligence. Its significance is who signed it. Leaders of three of the companies building the most advanced systems agreed in writing that extinction risk from AI deserved global attention.

What is the 2025 Statement on Superintelligence?

The Statement on Superintelligence is a one-sentence call for a prohibition, released by the Future of Life Institute on October 22, 2025, and aimed at superintelligence by name.

“We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in.”

— Statement on Superintelligence, Future of Life Institute, October 2025

Early signers included Hinton and Bengio, Apple co-founder Steve Wozniak, Richard Branson, former US National Security Advisor Susan Rice, retired Admiral Mike Mullen, former Irish President Mary Robinson, commentators Steve Bannon and Glenn Beck, and Prince Harry and Meghan, Duchess of Sussex. The list spans the political spectrum.

Signing is open to the public. On October 15, 2026, the statement’s website showed 77,577 signatures, including 5,000 from the same petition run by Ekō.

The institute released a survey of 2,000 US adults, conducted September 29 to October 5, 2025, alongside the statement. It reported that 64% of respondents said superintelligence should not be developed until there is scientific agreement that it is safe and controllable, or should never be developed, 73% supported robust regulation, and 5% supported the unregulated status quo. Neither the press release nor the results page names a polling firm.

The difference from 2023 is the ask. The 2023 statement asked for priority. The 2025 statement asks for a prohibition with conditions.

What do the skeptics argue?

Skeptics argue variously that superintelligence is poorly defined, that intelligence cannot run away, that it is not close, or that the word is marketing. These positions do not all agree with one another.

The concept is unclear

Arvind Narayanan and Sayash Kapoor of Princeton argue in “AI as Normal Technology” (April 2025) that superintelligence is “incoherent as usually conceptualized.” They treat AI as a powerful but ordinary technology whose effects will spread over decades, and they argue policy should build resilience rather than try to control a hypothetical superintelligence.

Intelligence does not explode

François Chollet argued in 2017 that intelligence is situational and limited by the environment, so a self-improving system would not run away. The intelligence explosion guide covers this in detail.

It is not close

Yann LeCun, then Meta’s chief AI scientist, told the Financial Times in May 2024 that large language models will never reach human-level intelligence, saying they cannot reason or plan as humans do. Coverage of his remarks at India’s AI Impact Summit in February 2026 ran under the headline “Superintelligence Isn’t Close.”

It is marketing

Some critics read company use of the word as branding. Fortune, reporting on Microsoft’s new superintelligence team in November 2025, called the rebranding “better marketing than science” and noted that no such systems exist.

Skeptics differ on whether superintelligence is impossible, distant or simply mislabeled.

What do the builders argue?

Companies pursuing superintelligence generally accept that it carries risk and argue that the benefits justify building it carefully.

• Dario Amodei’s “Machines of Loving Grace” (October 2024) predicts that powerful AI could compress 50 to 100 years of progress in biology and medicine into 5 to 10 years.
• Safe Superintelligence Inc. says it will advance capabilities as fast as possible while making sure safety stays ahead.
• Microsoft AI’s “humanist superintelligence” post argues for deliberately bounded systems that serve people, and rejects a race to AGI.
• Meta frames personal superintelligence as a way to put capability in individuals’ hands rather than in a central system.

The company definitions guide quotes each one.

What have governments done?

As of October 2026, governments have signed declarations and backed scientific reports on the risks of advanced AI, and lawmakers in the US and the UK have introduced bills to ban superintelligence. None of those bills has become law.

International declarations and reports

At the AI Safety Summit at Bletchley Park in November 2023, 28 countries and the European Union signed the Bletchley Declaration. It warned of the potential for “serious, even catastrophic, harm, either deliberate or unintentional” from the most capable AI models. New Zealand joined in October 2024.

The summit also led to the International AI Safety Report, a scientific review chaired by Yoshua Bengio and written by more than 100 experts. Its second edition was published in February 2026. It summarizes evidence and does not recommend policies.

United States

On September 23, 2026, Senator Bernie Sanders and Representative Greg Casar unveiled the Ban Artificial Superintelligence Act of 2026. Sanders introduced it in the Senate that day as S. 5493, and Casar introduced it in the House on September 24 as H.R. 10538. According to the Associated Press, it would permanently prohibit artificial superintelligence, pause the most advanced AI development until a new Department of Artificial Intelligence sets safety rules, and require federal approval before deploying advanced systems. The Senate text defines artificial superintelligence as a system that “exceeds human cognitive performance and capabilities across most domains or tasks,” or one with “sufficient capabilities to plan and execute the destruction or disempowerment of humanity,” including a system that can easily be modified to do either.

The Associated Press reported that the bill faces long odds in the Republican-controlled Congress. As of October 15, 2026, both versions were still in committee.

United Kingdom

On September 8, 2026, Labour MP Alex Sobel introduced a Ten Minute Rule bill in the House of Commons to prohibit the development, deployment and operation of artificial superintelligence. TIME reported that the campaign group ControlAI wrote the bill, and that Sobel called it the first bill seeking to ban superintelligence in any G7 parliament. The government has not backed it. Earlier, in January 2026, the House of Lords debated a question on superintelligent AI.

Neither bill uses Bostrom’s definition. The US bill has two tests: cognitive performance across most domains or tasks, and the capacity to destroy or disempower humanity. According to TIME, the UK bill defines superintelligent AI by its capacity to disempower state authorities. Under either bill, what a system could do, not only how intelligent it is, would help decide whether it is banned.

Where do the sides agree?

On several basic points:

• No current system meets the classic definition of superintelligence.
• The term is used loosely, including by companies.
• The core disagreement is about timing, controllability and whether to act before the evidence is in.