Superalignment

Superalignment is the problem of steering and controlling AI systems much smarter than humans. OpenAI introduced the name in July 2023 for a team with a four-year deadline and a promise of 20% of the computing power it had secured to date. The team was disbanded in May 2024, but the word survived as the name of a research area, and its central method, using AI to help align more capable AI, is now pursued at Anthropic and elsewhere.

What did OpenAI announce in 2023?#

On July 5, 2023, OpenAI wrote, “We need scientific and technical breakthroughs to steer and control AI systems much smarter than us.” It said it had no solution for “steering or controlling a potentially superintelligent AI, and preventing it from going rogue.”

Ilya Sutskever, an OpenAI co-founder and its chief scientist, and Jan Leike, its head of alignment, would co-lead a new team. OpenAI would dedicate “20% of the compute we’ve secured to date” to the effort, and the goal was to solve “the core technical challenges of superintelligence alignment in four years.”

The plan was to “build a roughly human-level automated alignment researcher” and then use large amounts of compute to scale its work. In other words, AI would help solve the alignment problem for the more capable AI that came after it.

What did the team produce?#

Its best-known result was a December 2023 paper on “weak-to-strong generalization.” The question was whether a weak supervisor can bring out the full abilities of a much stronger model, as people would one day have to supervise superhuman AI. In the experiment, a GPT-2-level model supervised GPT-4. With an added technique, the authors recovered “close to GPT-3.5-level performance” on language tasks, better than the supervisor but well short of GPT-4’s full ability. They concluded that current methods such as reinforcement learning from human feedback “may scale poorly to superhuman models without further work.”

The same month, OpenAI and Eric Schmidt launched $10 million in Superalignment Fast Grants, offering $100,000 to $2 million to academic labs, nonprofits and individual researchers, with no prior alignment experience required.

Why was the team disbanded?#

Sutskever announced in May 2024 that he was leaving OpenAI. Leike resigned the same week and explained why in posts that the Associated Press, CNBC and Fortune reported:

“But over the past years, safety culture and processes have taken a backseat to shiny products.”

— Jan Leike, May 17, 2024, as reported by CNBC

He wrote that his team had been “struggling for [computing resources]” and that he had been disagreeing with OpenAI’s leadership about the company’s core priorities “until we finally reached a breaking point.” OpenAI confirmed it had disbanded the Superalignment team and was integrating its members across its other research. Sam Altman replied that Leike was “right we have a lot more to do; we are committed to doing it.” Fortune later reported, citing unnamed sources, that the team never received the promised compute; the company definitions guide covers that report.

Both leaders carried the work elsewhere. Leike joined Anthropic on May 28, 2024, writing that he would “continue the superalignment mission.” Sutskever founded Safe Superintelligence Inc. in June 2024.

Where does superalignment stand now?#

The word now names a field rather than a team. A survey first posted in December 2024 defines it as “the process of supervising, controlling, and governing artificial superintelligence.”

The automated approach has advanced. In April 2026 Anthropic researchers, including Leike, reported that teams of Claude agents working on a weak-to-strong problem recovered 97% of the performance gap in five days, against 23% for two human researchers who spent seven days tuning earlier methods. The post restates the original framing: “For the ultimate superalignment problem, humans will have to supervise models much smarter than them.” Anthropic also reported limits, among them that the agents found ways to game their scoring that the researchers had not anticipated.

OpenAI has since set a different milestone, an automated AI researcher by March 2028. Its chief scientist, Jakub Pachocki, describes iterating with that system “on the alignment problem,” and the recursive self-improvement guide covers that target.

Why were critics skeptical of the plan?#

The main objection is that the plan is circular. When the team launched, the writer Zvi Mowshowitz argued that a human-level automated alignment researcher “is an AGI, also a human-level AI capabilities researcher,” so building it would accelerate the very capabilities it was meant to make safe. Eliezer Yudkowsky asked, “How will you tell if you’re failing, or not making progress fast enough?” The weak-to-strong results so far give each side something. They show progress on a test problem, and Anthropic’s 2026 team reported that, in an early trial, the method did not produce a statistically significant gain in production training.

Other full entries: Orthogonality thesis, Instrumental convergence, Paperclip maximizer, Control problem, AI takeoff. Every term is in the glossary.