Attention: The Mechanism That Rewired Everything

In 2014, neural machine translation was a hot problem. Recurrent neural networks could translate sentences, but they struggled with long ones. The reason was architectural: RNNs compress the entire source sentence into a single fixed-size vector, a bottleneck that lost information when sentences grew complex. A 50-word French sentence was squeezed through the same-sized doorway as a 5-word one.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio proposed an elegant fix: instead of making the decoder rely on a single compressed vector, let it look back at all the encoder's hidden states and decide, for each word it generates, which parts of the input sentence are most relevant. The decoder learns to "attend" to different source positions at different steps. Translating "le président" in a long French sentence, it learns to attend heavily to the words around the subject, not the whole sentence equally.

The mechanism worked dramatically well on long sentences. More interestingly, attention weights could be visualised. You could see exactly which source words the model was attending to when generating each target word. This transparency was new. Neural networks had always been black boxes, but attention gave you a window.

For three years, attention was an add-on, an improvement to RNNs, a way to make sequence-to-sequence models better. Then in 2017, Vaswani et al. published "Attention Is All You Need" and removed the RNN entirely. Attention alone, no recurrence, no convolution, turned out to be sufficient. The Transformer architecture was born, and with it: BERT, GPT, and every large language model that followed.

Bahdanau et al. titled their paper simply "Neural Machine Translation by Jointly Learning to Align and Translate." They were trying to translate sentences. They accidentally invented the foundation of modern AI. The best mechanisms tend to be like that: they solve one small problem precisely, and the precision turns out to be universal.