TD-Gammon: The Neural Network That Taught Itself to Play

In 1992, Gerald Tesauro at IBM Research published a paper describing a program called TD-Gammon. It had played 1.5 million games of backgammon, against itself. Starting with essentially random play, it improved steadily through a reinforcement learning algorithm called TD-lambda, adjusting its neural network weights based on the difference between predicted and actual outcomes. After 1.5 million self-play games, it played at the level of the world's best human backgammon players.

This was remarkable for several reasons. Backgammon is a game of partial information and chancedice introduce randomness that chess does not have. The state space is enormous. And yet a neural network trained purely on self-play, with no human games provided, no expert knowledge encoded beyond the rules, reached expert level. The program hadn't been taught to play. It had learned to play by playing.

TD-lambda, the algorithm TD-Gammon used, was based on temporal difference learning, a form of reinforcement learning where you update predictions based on the difference between what you expected and what you got, propagated backward through time. The theory had been developed by Richard Sutton and Andrew Barto throughout the 1980s, drawing on psychological models of animal conditioning. Pavlov's dogs and neural networks were doing the same thing.

TD-Gammon was a proof of concept for self-play reinforcement learning that influenced the entire subsequent field. When DeepMind began work on AlphaGo in 2014, Tesauro's paper was on their reading list. The idea that a program could teach itself through self-play, without human games, without hand-crafted heuristics, was TD-Gammon's gift to the field.

The program also discovered novel backgammon strategy. Human experts, studying TD-Gammon's play, found that it preferred certain opening moves they had considered suboptimal. When they tested those moves in practice, they found TD-Gammon was right. A neural network trained on self-play had discovered things that 5,000 years of human backgammon hadn't. This pattern, AI revealing that human conventional wisdom had blind spots, would recur dramatically with AlphaGo in 2016.