From magnets to the mind — How Hopfield and Hinton bridged physics and AI

Apart from showing how the boundaries between physics and engineering are blurring, the Nobel prize seems to be a nod towards science finally breaking down the millennia-old division between physical and living systems.

The award of the Nobel Prize in Physics to John Hopfield and Geoffrey Hinton ‘for foundational discoveries and inventions that enable machine learning with artificial neural networks’ has seen a level of public discussion about its appropriateness, possibly not seen since the award of the Nobel Prize for literature to Bob Dylan in 2016 made many doubt whether ‘pop songs’ constitute literature.

While Hopfield and Hinton are both well-respected figures in the international scientific community whose contributions have been long recognised by their peers, people seem to be puzzled as to how their work falls into the ambit of ‘physics’. A possible answer is that, apart from the boundaries between physics and, say, engineering, being somewhat arbitrary (in any case, they did not exist as separate fields even a couple of centuries back), the prize seems to be a nod towards science finally breaking down the millennia-old division between physical (or ‘inanimate’) and living systems.

This endeavour can be traced back to the 1940s, when the founders of quantum mechanics Niels Bohr and Erwin Schrodinger, on the one hand, and neurophysiologist Warren McCulloch working together with the gifted young mathematician Walter Pitts, on the other, had first indicated that aspects of both life and mind can be understood in terms of simple physical models.

The following decades have made it increasingly clear that processes in living organisms can be completely understood in terms of physics and chemistry, and does not require invoking any ‘vitalism’-like property that is supposedly unique to living systems. This view that life and mind are processes that emerge purely as a result of interaction between inanimate components is illustrated by the elegant work of Hopfield that was later extended by Hinton.

Starting from a model of a neuron treated as a simple ON/OFF switch, processes in the brain can be reproduced by a large number of such ‘neurons’ connected to each other via synapse-like links. The strength of each link is determined using a ‘learning rule’ proposed by the neuropsychologist Donald Hebb, stated simply as ‘neurons that fire together, wire together’. Using this setup, Hopfield’s landmark paper of 1982 associated an energy with every brain state — defined as a particular configuration of neurons that are ON (‘firing’) or OFF (‘quiescent’).

The initial state from which the ‘brain’ starts has a higher energy, which gradually comes to rest at the closest available minimum-energy state, viz., the memory learnt by Hebb’s rule. To see how, consider swinging a pendulum. By giving it an initial jerk, you are supplying some energy which makes it swing from side to side — i.e., you begin with a high-energy state. However, over time, friction will make the energy dissipate into the surroundings and the pendulum will gradually come to rest at the state having the lowest energy.

Hopfield further showed that by mapping the ON/OFF neurons to tiny magnetic spins that are oriented either UP or DOWN, this description of the brain is essentially equivalent to random magnets (technically called ‘spin glasses’) that physicists have been studying for some time. Thus, changes in the brain state can be described in terms of an energy landscape, with the memory states being the ‘valleys’ (low energy states) that are separated from other memory states by higher energy ‘ridges’. Each memory has a ‘basin of attraction’ corresponding to the sensory stimuli that trigger the stored memory. Thus, when we smell freshly baked bread, and it gives rise to a Proustian chain of thoughts that make us remember an event in our childhood, e.g., grandmother baking in the kitchen, this is nothing but an initial higher-energy brain state (smelling the bread) gradually flowing down to the neighbouring lowest energy state (the memory of our childhood).

Showing that memory storage/recall in the brain can be described via an energy landscape, very soon made physicists enter brain science in droves — dubbed the ‘Hopfield revolution’ — where they solved problems that previously couldn't even be quantified, like how does memory capacity change with brain size.

Hopfield’s model however had severe limitations. The brain has a hierarchical structure, as sensory neurons respond to external stimuli, which then activate layers of interneurons one after another, before eventually causing motor neurons to activate muscles. Geoffrey Hinton working with Terrence Sejnowski realised that not all neurons in the Hopfield model need to be involved in the reception of stimuli and/or readout of memory.

By dividing neurons into ‘visible’ (whose states encode the memorised pattern) and ‘hidden’ (which perform computations but are not directly accessible for input or output), and by allowing probabilistic evolution of the states (to take into account the fluctuations inherent in processes going on inside the brain) Hinton came up with the Boltzmann machine model. Subsequently, Hinton combined a layered architecture, where connections exist only between successive layers, with the energy landscape framework of Hopfield, to develop restricted Boltzmann machines.

This allowed networks with arbitrary numbers of layers to be trained efficiently by altering their connection weights to perform any given function. The resulting birth of the deep learning paradigm is — as we are all well aware — radically transforming society through the AI revolution.

(Sitabhra Sinha is Professor of Physics and Dean of Computational Biology Graduate Program, The Institute of Mathematical Sciences, Chennai.)

Disclaimer: The views expressed above are the author's own. They do not necessarily reflect the views of DH.