Luca Chiodini presenting “The Illusion of Thinking?” at the USI Software Institute.

Recorded Talk

Video on YouTube

Abstract

How do we teach language models to “think”? Over the last couple of years, researchers have experimented with prompting strategies such as chain-of-thought and self-verification, as well as model fine-tuning (e.g., via reinforcement learning). Combined, these techniques gave rise to a class of language models dubbed “thinking” or “reasoning” models. These advanced models have exhibited surprising capabilities to solve problems that eluded classical language models. Scientific debate is still ongoing to determine whether models of this class are able to genuinely reason. This talk will present a recent paper by researchers at Apple (“The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity”) which compares reasoning and classical models when solving different tasks at different complexity levels. The results offer a few surprises: classical models outperform reasoning models on low complexity tasks, and reasoning models fail to solve tasks at high complexity even when provided an explicit algorithm in pseudocode. Are reasoning models actually thinking, or is it all a sophisticated illusion?