The Vanishing Gradient: Why Deep Networks Couldn't Learn, Until LSTM

Backpropagation had solved training for shallow networks. But researchers trying to go deeper, 10, 20, 50 layers, kept hitting a wall. Networks simply wouldn't learn. Early layers received near-zero gradient updates while later layers trained reasonably well. The useful training signal evaporated as it was propagated backward through many layers. This was the vanishing gradient problem.

Sepp Hochreiter identified it precisely in his 1991 diploma thesis at the Technische Universität München, under Jürgen Schmidhuber. His analysis showed why: in networks with many layers and sigmoid or tanh activations, gradients were repeatedly multiplied by numbers less than 1 as they propagated back. After enough layers, they vanished to effectively zero, no signal, no learning. The mathematics was clear; the solution was not.

In 1997, Hochreiter and Schmidhuber published the Long Short-Term Memory (LSTM) network. The key innovation was a "memory cell," a unit with controlled read, write, and erase operations governed by learned gates. Because information could flow through the cell unchanged (multiplied by values near 1, controlled by the forget gate), gradients could propagate back across long sequences without vanishing. LSTMs could learn dependencies spanning hundreds of time steps.

LSTMs became the dominant architecture for sequence modelling throughout the 2000s and early 2010s. Speech recognition, machine translation, text generation, time series prediction, any problem where order mattered and long-range dependencies existed. Apple's Siri, Google's Smart Reply, Amazon's Alexa, all used LSTM-based models in their early production systems.

The vanishing gradient problem was eventually addressed more broadly by architectural innovations: residual connections (He et al. 2015), which added "skip connections" that allowed gradients to bypass layers, enabled training of networks with hundreds of layers. And then the Transformer, which avoided the problem structurally through attention. But Hochreiter's 1991 identification, and the LSTM that followed, kept sequential learning alive through a decade when it would otherwise have been impossible.