Why a Computer Scientist Thinks Superintelligence Won’t Happen

Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. It started when Anthropic researcher Jacob Coxon announced his resignation, warning that his employer and OpenAI were “racing straight to self-improving superintelligence and gambling with our lives.” Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

These concerns stem from a belief that has circulated in futurist circles for many decades: Once engineers design an AI system capable of improving itself, it will trigger an accelerating cycle of autonomous upgrades known as recursive self-improvement (RSI). The result will be artificial superintelligence (ASI), a system with cognitive capabilities so vastly exceeding those of humans that it will easily evade our control and perhaps even wipe out our species. Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

These ideas have prompted varied reactions. Some argue that the concept of ASI is being used for marketing or to instigate regulation that will help the incumbent labs squash new competitors. Others focus on the plausibility of such systems killing us all, debating the finer points of specific scenarios such as AI-engineered superviruses. Still others claim that aligning ASI to human interests is feasible and so we should rush ahead to usher in this digital utopia.

Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible? The answer is important because it can help us calibrate our anxieties and develop the most effective possible responses to this emerging technology.

I recently explored this topic in a computer science doctoral seminar that I taught at Georgetown University, in Washington, D.C., in which my students and I examined the technical and cultural roots of the superintelligence story. (For those who are interested, we published an annotated bibliography of our class readings.) I came away with several key objections to the feasibility of ASI, which I’ll detail below, as well as a deeper fear that even if the promised intelligence explosion never arrives, pursuing it can still create real harm.

Objection #1: Recursive Self-Improvement Is Speculative

RSI is an old concept. In a 1965 paper, statistician I.J. Good speculated that a rapid cycle of AI self-improvement could lead to an “intelligence explosion.” The idea gained traction in the 1990s among the Extropians, a Bay Area futurist group, where it offered a way to discuss ASI without having to address the engineering details of how one would build such a system. The concept was taken up by such notable figures as Eliezer Yudkowsky (who later founded the rationalism movement) and Nick Bostrom (who helped steer effective altruism to focus more on existential risks to humanity).

Recently, however, this topic has shifted from thought experiment to practical engineering within both Anthropic (makers of Claude) and OpenAI (makers of ChatGPT). In a June blog post titled “When AI Builds Itself,” Anthropic described how LLM-powered agents are speeding up the writing, debugging, and optimization of code, and also automating parts of research such as running experiments and logging results. The company’s internal data, the report concluded, “is showing that AI is already accelerating the development of AI systems.” OpenAI has been similarly explicit about its intentions; in a September report, the company announced a goal of developing an automated researcher by March 2028.

But do these efforts put the companies on a trajectory toward uncontrollable RSI? Several obstacles undermine this claim. Consider the availability of the data needed to improve AI systems. Through 2023, major advances in LLMs came from scaling up the size of the models. The original GPT-1 had 117 million parameters (values learned during training that shape how the neural network processes information). This quantity jumped to 1.5 billion for GPT-2, to 175 billion for GPT-3, to (likely) more than a trillion for GPT-4’s release three years ago. After GPT-4, however, continued scaling began yielding diminishing returns. This led engineers to increasingly turn their attention toward “post-training” phases, where techniques from reinforcement learning are applied to improve an already trained model’s performance on specific tasks. This approach rewards a model for producing successful responses, helping it better produce similar answers in the future. It requires, however, lots of examples of relevant inputs paired with correct outputs.

Such datasets are relatively easy to generate in highly structured fields such as computer programming and mathematics, which helps explain why we’ve seen rapid and impressive advances in these narrow domains. Progress in areas that lack easy access to this type of training data has been less dramatic. This might explain why AI can tackle advanced theorems but struggles to empty an email inbox. “What if the jagged frontier is mainly math + code (which you can push arbitrarily far with [reinforcement learning techniques]),” the French AI researcher François Chollet recently hypothesized, “and everything else starts to plateau because it is still bottlenecked by human generated data?” The deployment of automated coding agents and LLM-powered researchers won’t overcome this fundamental limitation.

The Game of Superintelligence

How a decades-old thought experiment became an AI-industry ambition—and where it might lead

Key

• Idea: A concept, argument, or vision

• Technical advance: A scientific or engineering development

• Reality check: A challenge to assumptions about AI progress

1. Alan Turing

1936

Introduces the concept of a universal computing machine. He later proposes that machines could think, laying the foundations for AI.

REALITY CHECK
Shows that some problems can’t be solved by any algorithm.2. I. J. Good

1965

Proposes that an intelligent machine could design an even smarter one, setting off a potential “intelligence explosion.”

3. Backpropagation

1986

Geoffrey Hinton, David Rumelhart, and Ronald Williams demonstrate how backpropagation can train multilayer neural networks.

4. Vernor Vinge

1993

Proposes that an intelligent machine could design an even smarter one, setting off a potential “intelligence explosion.”

REALITY CHECK
Notes that faster thinking alone isn’t enough for superintelligence: A ­faster-thinking dog still couldn’t play chess.5. Nick Bostrom & Eliezer Yudkowsky

1990s–2000s

Bostrom and Yudkowsky discuss superintelligence in Extropian circles, then spread the idea to effective-­altruist and rationalist circles, respectively.

6. Deep learning breakthroughs

2010s

Larger models, more data, and greater computing power produce rapid ­improvements in AI capabilities.

7. The race to self-improving AI

2020s

Dario Amodei and other AI leaders set their sights on superintelligence, with recursive self-improvement proposed as a path toward it.

8. Reality check

2020s

François Chollet questions whether LLMs’ current advances in math and coding will extend to domains where training data and reliable feedback are harder to obtain.

YOU ARE HERE

2026

AI systems are already accelerating AI development, and labs are working on automated researchers. But will this lead to an intelligence explosion?

THREE POSSIBLE FUTURES1. The Intelligence Explosion

AI systems improve themselves at accelerating speed, surpass human intelligence, and potentially escape our control.

2. Danger Without Superintelligence

The intelligence explosion never arrives, but increasingly autonomous AI systems cause real-world harm through misuse, poor design, or loss of human control.

3. A Useful AI Future

Powerful, specialized AI tools bring major benefits, with constraints that make their behavior easier to predict and control.

Claims that RSI is imminent also understate the massive scientific efforts required for major AI advances. Consider the critical role of backpropagation, an algorithm that enables very large neural networks to learn by tuning internal parameters, layer by layer, to reduce errors in their predictions. This algorithm enabled the deep learning revolution, but it didn’t emerge ex nihilo in the minds of its creators. The influential 1986 paper on backpropagation by David Rumelhart, Geoffrey Hinton, and Ronald Williams built on decades of critical breakthroughs in mathematics and computer science.

An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened. This would require AI agents to somehow simulate, organize, and evaluate decades of worldwide scientific effort across numerous fields—an impossibly ambitious task. Indeed, recent research led by computer scientists at Princeton found that current AI tools had difficulty making autonomous progress even on highly constrained research questions. “Our results provide early evidence that today’s agents can do the engineering of AI research,” the researchers conclude, “but struggle with critical parts of the research life cycle.” The leap from this current reality to a future in which these same types of tools are automating science is almost too massive to imagine.

Finally, the RSI narrative downplays resource limitations. Tens of thousands of human brains have worked for decades, aided by increasingly powerful computers, to develop today’s AI systems, which can match human intelligence in some

[…]
Content was trimmed to protect the source. Please visit the original article for the full text.

This article has been indexed from IEEE Spectrum