Do large language models truly think? This question has always lingered in people's minds.
As an opponent of LLMs, Yann LeCun has presented new evidence. His latest research, "From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning," uses a fresh perspective from information theory to reveal the fundamental differences between large language models (LLMs) and humans in "understanding the world."
When processing information, the human brain excels at compressing complex perceptions and experiences into concise and meaningful "concepts." For instance, "robin" and "blue jay" are both categorized as "birds," and we can also discern that a "robin" is more "bird-like" than a "penguin." This ability allows us to efficiently generalize while remaining sensitive to details and context when faced with vast amounts of information.
The "understanding" mechanism of LLMs, however, is significantly different. Through statistical learning from vast amounts of text, they form complex word embedding spaces. The paper's authors raise the question: Can the internal "conceptual structures" of these AI models also preserve rich semantics while compressing information, like humans do? Or are they merely "clever compressors," fundamentally different from human cognition?
A New Information Theory Framework: Quantifying the Gap Between AI and Humans Using Rate-Distortion Theory
To answer this question, the research team innovatively introduced "Rate-Distortion Theory" and "Information Bottleneck" from information theory, establishing a new quantitative framework. Simply put, this framework can precisely measure the trade-off a system makes between "compressing information" (reducing redundancy) and "preserving semantics" (avoiding distortion).
The researchers selected classic datasets from cognitive psychology (such as Rosch's "bird" and "furniture" typicality experiments), which were meticulously designed by experts to genuinely reflect human concept formation and "typicality" judgments. Concurrently, the team analyzed the word embedding structures of various mainstream large models, including BERT, Llama, Gemma, Qwen, Phi, and Mistral, covering scales from hundreds of millions to over seventy billion parameters.
Three Core Findings: The "Understanding Gap" Between AI and Humans
1. AI Can Learn to Categorize, But Struggles with Generalization
The study found that LLMs perform excellently in broad categorization, classifying "birds," "furniture," etc., with high accuracy. Some smaller models (like BERT) even outperform larger models in this regard. This indicates that AI can "understand" at a macro level which items belong to the same category. However, when it comes to more detailed "typicality" judgments, AI falls short. For instance, AI finds it difficult to perceive a "robin" as more bird-like than a "penguin," unlike humans. This lack of "fine-grained" semantic distinction means AI's "understanding" remains superficial. Such nuanced semantic distinction is a unique advantage of human cognition.
2. AI and Humans' "Optimization Goals" Are Fundamentally Different
The paper's greatest highlight is its revelation of the fundamental divergence between AI and humans in the "compression-meaning" trade-off. LLMs, in their internal representations, intensely pursue "compression"—expressing the most content with the least information, minimizing redundancy to the greatest extent. This "compression-first" strategy makes AI highly efficient in an information-theoretic sense, but it sacrifices sensitivity to semantic details and context. Human conceptual systems, conversely, prioritize "adaptive richness," meaning they retain more details and context, even if this reduces compression efficiency and requires more "storage space." This fundamental difference dictates the completely different ways both entities "understand the world."
3. Larger Models ≠ More Human-like; Structure and Objectives Are More Critical
The research also found that increasing model size does not necessarily bring large models closer to human thinking. Instead, the model's structure and training objectives are key factors influencing "human-like" performance. For example, encoder models like BERT sometimes outperform larger generative models on certain tasks. This finding challenges the current trend of AI "stacking parameters."
Summary
Technological progress is not just about "bigger" or "faster"; more importantly, it's about "more suitable." If we want machines to better serve humanity, perhaps we need to rethink what truly valuable "intelligence" is. We don't have to demand that machines think like humans; perhaps it is precisely these differences that make the combination of humanity and technology more interesting and full of possibilities.
Paper: https://arxiv.org/pdf/2505.17117
Reply "join group" on the official account to join the discussion group.