The Algorithm's Illusion, The Scale's Revolution

The Truth Behind AI's Progress: Algorithmic Sparks vs. The Brute Force of Scale

In the last few years, artificial intelligence has burst into our lives at an unprecedented pace. We marvel at the rapid iteration of model capabilities and have grown accustomed to attributing it all to continuous algorithmic breakthroughs.

In 2020, a landmark study from OpenAI provided the most counter-intuitive and stunning backlash for this impression. They discovered that between 2012 and 2019, the compute required to train a neural network to the same performance level had decreased by a factor of 44, equivalent to algorithmic efficiency doubling every 16 months. Stunning, right? But see this: in the meantime the compute has increased 300,000x!

What does this speed signify? OpenAI provided the answer with the chart below.

Algorithmic vs. Hardware Efficiency Contributions

Source: OpenAI, "AI and Efficiency", 2020

The conclusion from this chart is staggering. It demonstrates that when it comes to reducing model training time, algorithmic progress (the red part) has contributed more than the hardware advancements driven by Moore's Law (the light blue bar). But most importantly, the most progress is actually offered by spending/parallelization (the purple part). This appeared to be an irrefutable declaration: in the development of AI, clever minds are far less important than powerful machines.

It was a perfect story of brute force triumphing over wisdom. It wasn't the picture of brilliant minds constantly devising more elegant algorithms, driving the exponential growth of AI like a perpetual motion machine that many hoped.

Moreover, a recent in-depth study from researchers at MIT and other institutions has posed a fundamental challenge to this illusion. It invites us to place that red section from the OpenAI chart under a microscope and see what's really hidden inside.

When we do, a clearer picture emerges.

The "Leverage" of Scale: The True Engine of Efficiency Gains

The MIT researchers conducted what could be described as "algorithmic archaeology." They found that a staggering 91% of the incredible 22,000-fold increase in AI training efficiency over the past decade did not come from minor, independently effective algorithmic tricks. Instead, it originated from two fundamental, "scale-dependent" shifts.

This means the advantages of these algorithms can only be truly "activated" after investing massive computational resources.

The first, and most significant, shift was the leap from the LSTM architecture to the Transformer architecture.

We can understand the difference using a concept from finance—"operating leverage":

• LSTM: This is like a traditional, labor-intensive factory. It runs well and has manageable costs at a small scale. But when orders (computational scale) grow exponentially, its internal management complexity skyrockets, and efficiency quickly hits a bottleneck.
• Transformer: This is like a highly automated, modern factory. Its initial fixed costs are extremely high, and it might even be less efficient than the old factory for small production runs. However, it possesses immense operating leverage. Once it crosses a certain scale threshold, the more resources you pour in, the lower its unit production cost becomes, and its efficiency grows explosively.

The chart below illustrates this visually.

Scaling Curves of Modern Transformers vs. LSTM

Source: Arxiv, On the Origin of Algorithmic Progress in AI

In panel (a), the green line representing LSTM is not far behind the purple line for Transformer at lower compute levels (the x-axis). But as compute extends to the right, the green line's descent slows noticeably. In contrast, the purple line continues to "consume" more compute, maintaining its steep downward slope. This is the most direct illustration of "scale-dependence."

The second key shift was an upgrade in our understanding of "Scaling Laws," specifically the cognitive shift from Kaplan's laws to Chinchilla's laws.

This is akin to a portfolio "rebalancing." The earlier Kaplan laws suggested that to make a model more powerful, one should primarily increase model parameters (like putting 80% of your capital into stocks). But the later Chinchilla research discovered that for optimal efficiency, model parameters and the amount of training data should be increased in nearly equal proportion (more like a 50/50 stock-bond balanced strategy).

Simply adjusting this "asset allocation" ratio yielded enormous efficiency dividends. Furthermore, this dividend also becomes more pronounced at larger scales.

Efficiency Difference Between Kaplan and Chinchilla

Source: Arxiv, On the Origin of Algorithmic Progress in AI

The True Value of "Minor Tweaks"

So, what about the algorithmic innovations we're all familiar with, like new activation functions (SwiGLU), normalization techniques (RMSNorm), or optimizers (AdamW)? Where do their contributions lie?

Through "ablation experiments"—sequentially removing these innovations one by one from a baseline model—the MIT study provides a sobering answer. These "scale-invariant" improvements, while effective, offer mostly limited efficiency gains, many providing less than a 1.5x improvement. More importantly, their effects are not simply cumulative. The actual combined effect of packaging all these improvements together is far less than the product of their individual effects.

The Multiplier Effect of Algorithmic Improvements

Source: Arxiv, On the Origin of Algorithmic Progress in AI

It's like fitting a race car with better windshield wipers or more aerodynamic side mirrors. These improvements are helpful, but they are not the core factor determining the car's lap time. The decisive factor is that we swapped the piston engine for a jet engine.

The "Staircase" Model of AI Progress

Synthesizing these points, a new picture of AI progress emerges. This decomposition plot from the MIT study serves as the ultimate deconstruction of OpenAI's grand bar chart.

Decomposition of Algorithmic Progress

Source: Arxiv, On the "Origin of Algorithmic Progress in AI"

In this chart, as the compute scale (horizontal axis) increases to the right, the total efficiency gain (vertical axis) climbs steadily. The scale-dependent innovations—represented by the dark blue (Kaplan scaling law) and light pink (Chinchilla scaling law)—occupy the vast majority of the area, and their contribution expands dramatically as scale increases. Meanwhile, the thin bands representing all other scale-invariant innovations (like MoE) remain compressed into a thin layer.

This chart tells us clearly: the AI "explosion" we've witnessed in recent years is largely because we happen to be on a steep "staircase step" initiated by the Transformer architecture, and we have been relentlessly sprinting to the right along the x-axis of compute scale, fueled by massive capital expenditure.

AI's progress is more like a "staircase":

• The "Steps": These are composed of a few fundamental, scale-dependent paradigm shifts. These leaps must wait for hardware capabilities to become powerful enough to support a new, larger scale of computation. When a new generation of hardware (like NVIDIA's latest GPUs) arrives and allows compute to cross a certain threshold, AI capabilities take a major jump.
• The "Landings": Between these major steps are relatively flat plateau periods. During these phases, researchers refine the existing paradigm with various minor algorithmic tweaks and optimizations. These efforts are important, but they cannot deliver exponential performance breakthroughs on their own.

Implications for Investment and the Future

This new understanding of AI progress holds crucial implications for how we observe the industry and make investment decisions:

• Compute as the Moat: Since the vast majority of algorithmic dividends are strongly tied to scale, the real moat is no longer a specific, unique algorithm. It is the capital expenditure (CapEx) and engineering prowess required to support massive scale. This explains why the AI race is increasingly a game played by giants like Google, Microsoft/OpenAI, and Anthropic.
• Beware the Arrival of a "Plateau": The "low-hanging fruit" of the LSTM-to-Transformer transition has already been picked. Until the next architecture capable of altering the slope of the scaling laws emerges, the pace of AI progress may slow considerably, entering a prolonged "plateau phase."
• Hardware Cycles as a Key Leading Indicator: The rhythm of AI progress is now tightly bound to the hardware release cycle. The next great leap in AI capabilities is likely gestating in the labs of NVIDIA and TSMC. Consequently, attention to the hardware industry has never been more critical.

Ultimately, the development of AI may not be the smooth, upward-trending exponential curve we once imagined. It more closely resembles a revolution punctuated by intermittent bursts, one founded upon the "brute-force aesthetics" of massive scale expansion. And the spark of algorithmic genius? It serves as the indispensable torch that illuminates our path forward during the lulls between these revolutions.

On the Origin of Algorithmic Progress in AI

Mascot Sammy6