Click “Turing AI” above and select “Star” public account
The AI dry goods you want to know, delivered to you as soon as possible
Copyright Statement
Reprinted from Jizhi Club, copyright belongs to the original author, for academic sharing. If there is any infringement, please leave a message to delete.
Artificial intelligence is changing the world at an unprecedented pace, but its core mechanisms extend far beyond complex algorithms and computational stacking. This article starts from the research of neuroscience pioneer John Hopfield, tracing the development of deep learning and revealing an unexpected fact: the theoretical foundation of many modern AI models originates from the “spin glass” model proposed by physicists in the last century when studying magnetic materials. These concepts, derived from statistical physics, not only explain how neural networks “remember” but also predict how they “create” when facing massive data. When AI exhibits behaviors beyond its designed expectations, we may be witnessing the emergence phenomenon. Understanding all of this may be a crucial step towards explainable AI.
Keywords: Hopfield Network, Diffusion Model, Spin Glass, Energy Landscape, Emergence, Statistical Physics, Deep Learning, AI
Zhao Siyi | Translator
Zhou Li | Reviewer
Article Title: The Strange Physics That Gave Birth to AI
Article Address: https://www.quantamagazine.org/the-strange-physics-that-gave-birth-to-ai-20250430/
Source: Quantamagazine
Spin glasses may be the most useful “useless thing” in history. Although called “glass,” these materials are mostly metallic. In the mid-20th century, their perplexing behavior attracted the attention of a small group of physicists. As materials, spin glasses themselves had no practical applications. However, the theories developed to explain their peculiarities ultimately triggered today's artificial intelligence revolution.
In 1982, condensed matter physicist John Hopfield drew upon the physical principles of spin glasses to construct simple networks that could learn and remember. This work rekindled interest in neural networks — these web-like “artificial neurons,” which had almost been abandoned by AI researchers at the time — and introduced physics into a new field: the study of the mind, both biological and mechanical.
Hopfield viewed “memory” as a classic collective physics problem in statistical mechanics: how does a system composed of multiple parts evolve? For any simple physical system, including spin glasses, thermodynamics tells us the answer is: “tends towards a lower energy state.” Hopfield utilized this simple property of collective phenomena to find a way to store and recall data using “artificial neural networks.” Simply put, he found a method to “place” memories at the bottom of an energy valley. The resulting Hopfield network, therefore, does not need to look up information; it simply “slides down the slope” to complete memory retrieval.
“The Hopfield network was a ‘conceptual breakthrough,’” said Marc Mézard, a theoretical physicist at Bocconi University in Milan, Italy. With the physics of spin glasses, later AI researchers could “use a whole set of tools developed for these old physical systems.”
In 2024, Hopfield and AI pioneer Geoffrey Hinton were awarded the Nobel Prize in Physics for their work in the statistical physics of neural networks. This award surprised many, with some complaining that it seemed more like a victory for AI research than an honor for physics. But when the physics of spin glasses was used to model memory and build thinking machines, it did not lose its physical essence. Today, some researchers believe that the same set of physical principles Hopfield used to make machines “remember” can also be used to help machines “imagine” and design neural networks that we can truly understand.
Emergent Memory
Hopfield began his career in the 1960s, studying the physics of semiconductors. But by the late 1960s, he wrote:
“In condensed matter physics, I could no longer find problems suited to my strengths.” (Quoted from his 2018 essay [1])
So he began to look for new directions. A brief foray into biochemistry led him to propose a theory of how biological organisms “proofread” biochemical reactions [2], and he subsequently turned his attention to neuroscience.
“I was looking for a problem (a big problem), not an ordinary ‘problem,’” he recalled in his essay, emphasizing the need to find truly important issues. “For me, ‘how does the mind emerge from the brain’ is the most profound question humans have asked. This is undoubtedly a problem.”
Hopfield realized that associative memory was part of this big problem — and the tools he had accumulated from condensed matter physics could be used to solve it.
In ordinary computers, data is statically stored and accessed by address. This address is unrelated to the stored information itself; it is merely an access code. Therefore, if the address is even slightly incorrect, you will read wrong data. But human memory doesn't seem to work that way. We often remember things through association. A clue, a vague memory, can evoke an entire memory. For example, you smell the scent of lilacs and suddenly recall a childhood scene in your grandfather's garden; or you hear the first few lines of a song and suddenly find yourself singing the whole love song you thought you had forgotten. Hopfield spent years studying associative memory and converting it into neural network models. He experimented with various randomly connected neural networks and other possible memory models. It didn't go smoothly at first, until eventually, he discovered an unexpected key to solving this problem.
Spin Glass
In the 1950s, scientists studying certain dilute alloys (such as iron in gold) discovered that these materials exhibited strange phenomena. Above a certain temperature, these alloys (like aluminum) behaved like ordinary materials. They were not magnetic themselves but showed a weak response to external magnetic fields. For example, a very strong magnet could move an aluminum can, but aluminum itself could not be used as a magnet. Typically, materials like aluminum lose their magnetism immediately after the external magnetic field is removed. But when the temperature drops below a certain critical value, spin glasses behave differently — their instantaneous magnetization state is retained to some extent (albeit at a lower value). This is just one of the strange behaviors of spin glasses; their thermal properties are equally puzzling.
Around 1970, condensed matter physicists began to develop a theoretical understanding of these materials by adjusting the classic model used in physics to study collective magnetic behavior — the Ising model. The Ising model looks like a simple grid of arrows, each arrow pointing up or down. Each arrow represents an atom's intrinsic magnetic moment, or “spin.” This is a simplified description of a real atomic system, but by adjusting the rules of interaction between neighboring spins, the model can produce surprisingly complex behavior.
Generally, when adjacent arrows point in the same direction, the energy is lower, and when they point in opposite directions, the energy is higher. If spins can flip, the system state in the Ising model evolves towards a lower energy aligned state, just like a small ball rolling downhill. Magnetic materials like iron will eventually settle into a simple state where all spins are either all up or all down. This is vastly different from ferromagnetic systems (like iron), which eventually freeze into only two ordered states (all up or all down), whereas the spins of non-ferromagnetic systems always fluctuate randomly and do not stabilize. In spin glasses, randomness is frozen.
The Ising model is essentially a “toy model,” and using it to predict the behavior of real materials is a bit like planning surgery with a stick figure drawing. But miraculously, it often works. Today, the Ising model has become a staple tool in statistical mechanics. Its various variants appear in almost every corner of complex collective phenomena research — including memory research extended by Hopfield's work.
Spin Memory
From a simple perspective, the interactions between neurons and the behavior of magnetic spins in the Ising model share many similarities. First, neurons are often modeled as binary on-off switches: either firing signals or not firing. Spins also have two states: up or down. Furthermore, the firing of one neuron can promote or inhibit the firing of neighboring neurons. These variable interactions between neurons are like the variable spin-spin interactions in spin glasses. Lenka Zdeborová, a physicist and computer scientist at EPFL in Lausanne, Switzerland, said:
“Mathematically, one can replace what originally represented spins or atoms, and other systems can also be described with this toolkit.”
To build the network, Hopfield started with a network of artificial neurons, which could be in an “on” (firing) or “off” (resting) state. Each neuron influenced the state of all other neurons, and this influence could be adjusted. At any given moment, the network's state was defined by which neurons were firing and which were resting. You could encode this state in binary: firing neurons represented by 1, resting neurons by 0. Writing out the entire network's current state would be a string of bits. This network was not “storing” information; it was the information itself.
To “teach” the network a specific pattern, Hopfield “sculpted” its energy landscape by adjusting the strength of interactions between neurons, thereby placing the target pattern in a low-energy stable state. In this stable state, the network stops evolving and stably represents a pattern. He found a rule inspired by the classic neuroscience principle that “neurons that fire together, wire together”: if two neurons are both firing (or both resting) in the target pattern, strengthen their connection; if their states are inconsistent, weaken the connection. After such training, the network can “recall” this pattern again by simply “sliding down the slope” in its energy landscape to the bottom of an energy valley; it will naturally evolve to the equilibrium state corresponding to this pattern.
As theoretical physicist Mézard said:
“Hopfield established this connection, saying: ‘Look, if we can adjust the strength of connections between these neurons like we adjust spin glasses, maybe we can turn equilibrium points into memories.’”
Hopfield networks can remember multiple patterns, each corresponding to an energy valley. Which valley the network falls into depends on where it starts evolving. For example, in a network that remembers both “cat” and “spaceship” images, if the initial state roughly resembles a cat, it is more likely to slide into the “cat valley”; conversely, if the initial state contains geometric features of a spaceship, the network will tend towards the “spaceship valley.” This makes the Hopfield network a model of associative memory: given an incomplete or corrupted memory, it dynamically recovers the complete pattern.
Recommended Reading:
The renowned popular science magazine Physics Today published an article outlining the major contributions of Nobel laureates John Hopfield and Geoffrey Hinton, arguing that from the underlying principles of neural networks to the emergent capabilities of large models, physics is of great significance for understanding the underlying mechanisms of artificial intelligence, and applying physical thinking to real-world systems is expected to provide insights for breakthroughs in artificial intelligence.
The Physical Roots of Neural Networks: From Spin Glasses to Energy Landscapes | 2024 Nobel Prize
Old Models, New Ideas
Between 1983 and 1985, Geoffrey Hinton and colleagues further extended Hopfield networks. They introduced randomness, building a new type of neural network — the Boltzmann machine. This network no longer “remembered” specific patterns but learned statistical regularities in training data and could generate new data conforming to these regularities — an early generative AI model. In the early 21st century, Hinton used simplified Boltzmann machines to solve the problems that plagued deep neural network training, thereby driving the development of deep learning.
By 2012, deep neural networks developed by Hinton and other pioneers began to achieve breakthrough results in various fields, becoming impossible to ignore. “Everyone realized at the time: this is truly amazing and is completely transforming the entire tech industry,” said Zdeborová. Today, the generative AI models we use every day — such as large language models like ChatGPT and image generation models like Midjourney — are essentially based on deep neural networks, and their success can be traced back to the physicists in the 1970s who were unwilling to let the “strange behavior of spin glasses” be ignored.
However, Hopfield networks are not just an “old bridge” in AI development. As new ideas emerge, this old model is gaining new life.
In 2016, Hopfield and Dmitry Krotov of IBM Research discovered that the Hopfield network is not a single model but a broad class of models with different memory storage capabilities [3]. In 2020, Hubert Ramsauer's research team further pointed out that a key part of the Transformer architecture [4] used by most modern AI models is actually a member of this Hopfield network family.
Based on this discovery, Krotov and his team recently proposed a new deep learning architecture called the Energy Transformer [5]. Traditional AI architectures often rely on extensive trial and error for design, while Krotov believes that the Energy Transformer can systematically build AI models by purposefully designing their energy landscape, much like constructing a more complex Hopfield network.
Although Hopfield networks were originally designed for “memory,” researchers are now exploring their potential for “creation.” For example, the diffusion model behind image generators like Midjourney is inspired by the physical process of diffusion. During training, researchers add noise to image data (e.g., cat pictures) and then train the model to remove the noise. This is very similar to the function of Hopfield networks — the difference being that instead of returning to the same cat image, the diffusion model eliminates “non-cat” features from a noisy random initial state to generate a new cat.
Krotov and his colleagues (including Benjamin Hoover, Yuchen Liang, and Bao Pham) pointed out that diffusion models can actually be understood as a special type of modern Hopfield network [6]. This understanding can also be used to predict certain behavioral characteristics of such networks. Their research shows that inputting more and more data into a modern Hopfield network does not just saturate its memory capacity. Instead, the model's energy landscape becomes extremely rugged, and it eventually becomes more likely to “remember” a fictional memory than a real one — at which point it “becomes” a diffusion model [7].
For physicists, this phenomenon of qualitative change triggered by simple quantitative changes (e.g., increased training data) is not uncommon. As condensed matter physicist Philip Anderson said in 1972: “More is different.” [8] In collective systems, merely expanding the scale of interaction between components can lead to unexpected new behaviors. “The very fact that neural networks work is an emergent property,” Mézard said.
Whether it's deep learning architectures or the human brain itself, their ‘emergence’ is both fascinating and mysterious — we do not yet have a universal theory of emergence. Perhaps it is statistical physics — the earliest tool used to understand collective phenomena — that will not only help us use these complex artificial intelligence systems but also be key to understanding their essence.
References:
[1] Hopfield, John. Now What? 2018.
[2] Hopfield JJ. Kinetic proofreading: a new mechanism for reducing errors in biosynthetic processes requiring high specificity. Proc Natl Acad Sci U S A. 1974 Oct;71(10):4135-9. doi: 10.1073/pnas.71.10.4135. PMID: 4530290; PMCID: PMC434344.
[3] Krotov, Dmitry, and John J. Hopfield. “Dense Associative Memory for Pattern Recognition.” ArXiv:1606.01164 [Cond-Mat, Q-Bio, Stat], 27 Sept. 2016, arxiv.org/abs/1606.01164.
[4] Ramsauer, Hubert, et al. “Hopfield Networks Is All You Need.” ArXiv:2008.02217 [Cs, Stat], 28 Apr. 2021, arxiv.org/abs/2008.02217.
[5] Hoover, Benjamin, et al. “Energy Transformer.” ArXiv.org, 2023, arxiv.org/abs/2302.07253.
[6] Hoover, Benjamin, et al. “Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models.” ArXiv.org, 2023, arxiv.org/abs/2309.16750. Accessed 20 May 2025.
[7] Pham, Bao, et al. “Memorization to Generalization: The Emergence of Diffusion Models from Associative Memory.” OpenReview, 2024, openreview.net/forum?id=zVMMaVy2BY. Accessed 20 May 2025.
[8] Anderson, P. W. “More Is Different.” Science, vol. 177, no. 4047, 4 Aug. 1972, pp. 393–396, https://doi.org/10.1126/science.177.4047.393.
Featured Articles:
1.