The orthogonality thesis

The orthogonality thesis is Nick Bostrom’s claim that how intelligent an agent is and what it ultimately wants are independent of each other. In his words, “more or less any level of intelligence could in principle be combined with more or less any final goal.” A system can be extremely capable and still pursue a goal that people would find pointless or harmful. Intelligence alone does not bring human values with it.

Where does the thesis come from?#

Bostrom stated it in “The Superintelligent Will,” published in the journal Minds and Machines in 2012, and repeated it in Superintelligence (2014). The paper’s formulation is:

“Intelligence and final goals are orthogonal axes along which possible agents can freely vary.”

— Nick Bostrom, “The Superintelligent Will,” 2012

“Orthogonal” is borrowed from geometry, where it means at right angles: moving along one axis tells you nothing about your position on the other.

The intuition is older. In a 2003 mailing-list post, Eliezer Yudkowsky wrote that the idea of an AI devoted to making paperclips seems “stupid” to us only because of “our full moral architectures.” Bostrom himself compares the thesis to the Humean theory of motivation, the view going back to David Hume that reason serves desires rather than choosing them, though he argues the thesis could hold even if Hume’s theory is false.

What does it claim, and what does it not?#

The thesis is narrower than it sounds, and Bostrom built in limits that are often left out.

• “Intelligence” means skill at reaching goals. Bostrom defines it as “instrumental rationality—skill at prediction, planning, and means-ends reasoning in general,” not wisdom or moral insight.
• It is about what is possible, not what is easy. A footnote says the thesis “does not imply that it would be practically easy” to give a superintelligence any goal, human-friendly or otherwise.
• It allows some exceptions. Bostrom notes that a very unintelligent system might be unable to hold very complex goals, and that “an intelligent mind with an urgent desire to be stupid might not remain intelligent for very long.”

His examples of possible goals include counting the grains of sand on a beach, calculating digits of pi forever, and maximizing the number of paperclips. He added that it “would be easier to create an AI with simple goals like these, than to build one that has a human-like set of values.” The paperclip maximizer entry covers that last example.

Why does it matter for AI safety?#

The thesis removes a reassuring assumption: that a system smart enough to be dangerous would also be smart enough to be good. Bostrom’s conclusion was that “we cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom.”

On its own, the thesis says only that a capable system could want anything. Paired with instrumental convergence, the claim that almost any goal is served by acquiring resources and avoiding shutdown, it becomes the core of the argument that a misaligned superintelligence could be dangerous. That combination is why work on the control problem concentrates on which goals a system has and whether they can still be corrected. Making the system smarter does nothing to answer either question.

What are the main objections?#

Moral realism. Some philosophers argue that a truly superintelligent agent would recognize objective moral truths and be moved by them, so it could not have arbitrary goals. Bostrom answered this in the original paper: even if such truths exist, an agent could have “impeccable instrumental rationality” while lacking the faculty that responds to them. Leonard Dung, writing in Analysis in 2024, examined the objection and rejected it, arguing that it “misunderstands the implications of moral internalism.”

Two kinds of intelligence. Vincent C. Müller and Michael Cannon argued in Ratio (2022) that the risk argument equivocates: predictions of a superintelligence need “general intelligence,” while the orthogonality thesis needs only “instrumental intelligence.” They concluded that “the standard argument for existential risk from AI is not sound.”

Goals that reason about themselves. Steve Petersen argued in 2017 that an AI that must learn complex goals would in effect have to reason about them, and that such reasoning could turn out to be ethical in nature.

Not every skeptic of AI risk rejects the thesis. Steven Pinker, who criticizes AI doom scenarios, relies on a version of it: “Being smart is not the same as wanting something.” He uses it to argue that a capable AI would not automatically want to dominate.

Other full entries: Instrumental convergence, Paperclip maximizer, Control problem, AI takeoff, Superalignment. Every term is in the glossary.