The inventor of the bitter lesson has a bitter lesson for LLM scaling enthusiasts

The inventor of the bitter lesson has a bitter lesson for LLM enthusiasts. Other researchers agree.

In a recent post, Turing Award winner Richard Sutton says raw compute alone isn't enough for intelligence. AI science needs clever new methods and is "far from finished." His post sparked wide discussion in the AI community on X, a sign that Sutton struck a nerve.

In his famous 2019 essay "The Bitter Lesson," Sutton argued that general methods scaling with compute beat sophisticated human-engineered approaches eventually. What matters are ideas that grow with compute rather than rely on human assumptions. Ad

Sutton's position isn't new. In earlier statements, he criticized the limits of simply scaling large language models. He called synthetic data a "big mistake" because the real world is infinitely complex and any simulation of it is only "microscopic." Human expertise as a bottleneck prevents true scaling. Ad

Instead, he called for agents with world models that continuously learn from their own experience. The world is massively more complex than what internet text can capture, and today's language models don't keep learning after training, Sutton argues. What's needed is continuous learning without destroying old knowledge.

LLMs under scrutiny

AI researcher Jerry Tworek, who like many others left OpenAI, one of the major AI companies, to explore new AI approaches, puts Sutton's latest statement in context. The bitter lesson still holds, but the more compute goes into model training, the more the algorithm matters, Tworek writes. Ad

Some algorithms scale, others hit dead ends, and many dimensions of scaling remain completely unexplored. The temporary success of transformer scaling, combined with a misreading of the bitter lesson, "has reward-hacked the brains of a whole generation of scientists."

AI researcher Jayden Teoh adds another angle. He says he has repeatedly seen the bitter lesson hold back academic research, with promising ideas dismissed for straying too far from its spirit. Teoh argues that this is not Sutton’s fault but a misreading of his original message. Ad

Teoh also argues that ideas that seemed, in hindsight, not to follow the bitter lesson were often replaced by cleaner, more general approaches. That did not make those ideas wrong, he says; they were often what enabled progress at the time. He wants the research community to stop dismissing such work as soon as it appears. “An idea doesn’t necessarily have to survive scaling forever to be an important contribution to AI progress,” Teoh writes. Ad

The tension between Sutton's call for new learning methods and Tworek's criticism of blind transformer scaling shows up in a growing wave of startups. In the latest Frontier Radar #5, we tracked 45 new AI labs betting on exactly these gaps.

Tworek himself runs Core Automation, one of those labs, searching for learning algorithms beyond pretraining and reinforcement learning. What unites these founders, including Sutton with Oak Lab, is that they believe in much more capable AI but doubt today's recipe alone will get there.

Some still bet on LLM scaling

Prominent industry leaders see things very differently. OpenAI CEO Sam Altman considers large language model scaling far from exhausted and accuses LLM critics of slowing down an entire generation of researchers by being too sure about what scaling can't do.

As evidence, he points to an OpenAI model that recently disproved a mathematical conjecture mathematicians had long failed to crack. He considers world models useful, for instance in robotics, but says the data clearly favor further LLM scaling.

Anthropic CEO Dario Amodei argues similarly, calling scaling progress a rainbow with no end. As CEOs of the two AI companies whose entire technical approach rests on this bet, Amodei and Altman are obviously biased in its favor.

AI News Without the Hype – Curated by Humans