Backpropagation: The Algorithm That Would Not Be Published
Backprop was rejected by journals for years before it changed everything.
By the early 1980s, neural networks were stuck. The perceptron, a single-layer network, had been shown by Minsky and Papert to be fundamentally limited. Multi-layer networks could theoretically solve those limitations, but nobody knew how to train them. Updating the weights of hidden layers seemed impossible because there was no way to assign blame for errors to neurons you couldn't directly observe. This was the credit assignment problem.
David Rumelhart, Geoffrey Hinton, and Ronald Williams solved it. Their method, backpropagation, used the chain rule of calculus to propagate error gradients backward through the network layer by layer. Each neuron's weight was adjusted in proportion to how much it contributed to the final error. The idea wasn't entirely new. Paul Werbos had described a similar method in his 1974 PhD thesis, but Rumelhart, Hinton, and Williams implemented it clearly, demonstrated it worked, and connected it to cognitive science.
The trouble was getting it published. Their initial manuscript was rejected. Reviewers were skeptical, reviewers were dismissive, and the broader academic community was still deeply wary of neural networks after the AI winters of the 1960s and 70s. The paper eventually found a home in Nature in 1986, in an issue that also included work on immune system proteins and tectonic plate movements.
"Learning Representations by Back-propagating Errors" became one of the most cited papers in computer science history. It demonstrated that multi-layer networks could learn internal representations, not just memorise inputs, and could solve problems that single-layer perceptrons could not, including XOR. It cracked open the field.
Geoffrey Hinton would go on to spend the next three decades refining what backprop enabled: deep networks, dropout, RNNs, eventually the work that earned him a Nobel Prize in Physics in 2024. But in 1986, the paper that started all of it very nearly didn't get published. Sometimes the most important algorithms in history need someone stubborn enough to keep submitting.
What it actually meant
The real lesson of backprop was not the chain rule itself. Calculus had that since 1687. The lesson was that credit assignment through a computational graph was tractable. The barrier was conceptual: nobody believed you could train a multi-layer network until someone demonstrated that you could.
What enterprises get wrong today
Teams treat training as a black box and never instrument it. Models train silently with exploding gradients, collapsed activations, or catastrophic forgetting, and the team only finds out when production performance drops months later.
Related capabilities
More from Algorithms
Ask about this story
Backprop is one of the great persistence stories in sciencean algorithm that almost wasn't published, written by people who were told neural networks were dead ends. What do you want to know about it? The math, the history, or what it unlocked?