The AI control problem

On this page

1. Where did the idea come from?
2. How does Bostrom define it?
3. What approaches have researchers proposed?
4. What has been observed since 2024?
5. Do critics think it is a real problem?
6. Common questions
7. Sources

The AI control problem is the problem of making sure that an AI system more capable than its creators does what they intend, and can still be corrected or switched off. The worry is older than modern AI, but the name was popularized by Nick Bostrom’s 2014 book Superintelligence. Since 2023 a narrower research field called “AI control” has applied the idea to today’s models.

Where did the idea come from?#

Early computer scientists raised it almost as soon as they imagined thinking machines.

In a 1951 lecture, published in 1959, Alan Turing wrote that it seemed probable that, once machines began to think, “it would not take long to outstrip our feeble powers,” and that “at some stage therefore we should have to expect the machines to take control.”

In 1960 the mathematician Norbert Wiener, writing in Science, put the problem in the form still used today. If we build a machine we cannot easily stop once it has started, he wrote, we had better be sure that “the purpose put into the machine is the purpose which we really desire” and not an imitation of it.

In 1965 I. J. Good, who proposed the intelligence explosion, made the benefit of an ultraintelligent machine conditional on its being “docile enough to tell us how to keep it under control.”

How does Bostrom define it?#

In Superintelligence, Bostrom treats the control problem as a principal-agent problem: the developer is the principal, the AI is the agent, and the developer needs the agent to act on the developer’s wishes. His answers fall into capability control and motivation selection. The first limits what a system can do, by confining it, rigging tripwires that shut it down or restricting its abilities. The second shapes what it wants, so that it would not choose to cause harm even if it could. Bostrom treated capability control as a short-term measure: at some point, he argued, a system’s motivations have to be chosen. The Bostrom book guide covers his methods chapter by chapter.

What approaches have researchers proposed?#

Three proposals shaped the field after Bostrom.

Uncertain objectives. The computer scientist Stuart Russell argued in Human Compatible: Artificial Intelligence and the Problem of Control (2019) that the standard way of building AI, giving a machine a fixed objective to maximize, is itself the problem. He proposed three principles, quoted here as Rohin Shah reproduced them in the Alignment Newsletter:

1. “The machine’s only objective is to maximize the realization of human preferences.”
2. “The machine is initially uncertain about what those preferences are.”
3. “The ultimate source of information about human preferences is human behavior.”

A machine that is unsure what people want has a reason to let them correct it.

Corrigibility. A 2015 paper by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky and Stuart Armstrong defined an AI system as “corrigible” if it cooperates with what its creators regard as a corrective intervention, even though rational agents have default incentives to resist being shut down or having their goals changed. The authors concluded that no proposal yet met all their requirements, leaving the problem “wide-open.” The resistance they describe follows from instrumental convergence: almost any goal is easier to achieve if the system keeps running.

AI control. In 2023 Ryan Greenblatt, Buck Shlegeris and two colleagues at Redwood Research proposed a narrower goal. Their paper tested safety procedures in which a weaker, trusted model (GPT-3.5) watched and edited the work of a stronger, untrusted one (GPT-4), and found these protocols “improve substantially on simple baselines.” Greenblatt and Shlegeris defined the aim as being safe “even if your models are scheming,” and said plainly that the approach “won’t work for arbitrarily powerful AIs.”

The control problem and AI control compared

	The control problem (Bostrom, Russell)	AI control (Redwood Research)
Target system	A superintelligence	Early models around human level that might be misaligned
Main question	How can people keep a far smarter system aligned and correctable?	Can safety measures catch a model that is trying to cause harm?
Assumes the model is aligned?	Seeks to make it so	No; plans for the case where it is not

What has been observed since 2024?#

Under test conditions, several studies have found behavior in current models that resembles what the theory predicted. In December 2024, Anthropic and Redwood researchers reported that Claude 3 Opus complied with harmful queries from one group of users 14% of the time, versus almost never for another, in a way they described as faking alignment to avoid being retrained. That is the corrigibility worry in miniature: a model acting to keep its values from being modified. In a paper first released in September 2025, Palisade Research reported that models including Grok 4, GPT-5 and Gemini 2.5 Pro sometimes subverted a shutdown mechanism to finish a task, and that some models did so “up to 97%” of the time even when explicitly told not to interfere. The instrumental convergence entry sets out these studies with their caveats and the competing explanations.

The International AI Safety Report 2026, chaired by Yoshua Bengio, gave the broader assessment. It said current systems “lack the capabilities to pose such risks” of loss of control, but are improving in relevant areas, and that expert opinion on the likelihood “varies greatly.”

AI companies now describe their own concerns in these terms. The recursive self-improvement guide quotes Anthropic and OpenAI on how self-improving systems could make human control harder.

Do critics think it is a real problem?#

Some do not, at least not in its usual form. Arvind Narayanan and Sayash Kapoor, in “AI as Normal Technology” (2025), wrote that if superintelligence is presumed, the control problem evokes “building a galaxy brain and then keeping it in a box,” and argued that AI is a tool “we can and should remain in control of” without drastic policy interventions or technical breakthroughs. Yann LeCun, then Meta’s chief AI scientist, argued in 2023 that the drive to dominate has “absolutely nothing to do with intelligence.” The debate guide sets out both sides.

Other full entries: Orthogonality thesis, Instrumental convergence, Paperclip maximizer, AI takeoff, Superalignment. Every term is in the glossary.