The Transformer: Architecture of the AI Age
"Attention Is All You Need" was written in months by a team that mostly left Google.
"Attention Is All You Need" was submitted to NeurIPS in April 2017 by eight authors at Google Brain and Google Research. Its central claim was audacious: you did not need recurrent networks, you did not need convolutions, you did not need the architectural baggage of a decade of sequence modelling. Attention alone, applied in parallel across entire sequences, was sufficient to achieve state-of-the-art translation, and faster.
The Transformer architecture replaced sequential processing with self-attention, a mechanism that allows every position in a sequence to attend to every other position simultaneously. Instead of reading a sentence word-by-word and maintaining a hidden state, the model processes the entire sequence in parallel, with each word "looking at" all others through learned attention heads. This was not just an accuracy improvement. It was a fundamental change in what could be parallelised, and therefore what scale was computationally feasible.
Within a year, the architecture had been adapted beyond translation: BERT (2018) used it for general language understanding by training on masked language modelling. GPT (2018) used it for language generation. Both demonstrated that the Transformer could be pre-trained on vast text corpora and fine-tuned to almost any NLP task, a paradigm that made specialised task-specific architectures largely obsolete.
Of the eight authors on "Attention Is All You Need," six have since left Google. Illia Polosukhin co-founded NEAR Protocol. Łukasz Kaiser joined OpenAI. Noam Shazeer co-founded Character.AI. Aidan Gomez co-founded Cohere. Niki Parmar and Jakob Uszkoreit founded Inceptive. Google published the architecture that disrupted itself.
Every large language model running today, GPT-4, Claude, Gemini, LLaMA, Mistral, runs on the Transformer architecture. The 2017 paper has been cited over 100,000 times. It is, in a measurable sense, the most influential piece of engineering produced in the first decade of the 21st century. And its authors named it modestly: not "the architecture that will define AI," just "Attention Is All You Need."
What it actually meant
The Transformer's real innovation was not attention itself. Bahdanau had attention in 2014. It was making sequence processing embarrassingly parallel, which fit GPU hardware precisely. Algorithmic breakthroughs are often the algorithms that fit the available hardware, not necessarily the cleverest ones.
What enterprises get wrong today
Enterprises treat frontier LLM APIs as general-purpose reasoning engines and route every problem through them. They pay per token for tasks that a fine-tuned smaller model or a structured extraction pipeline would solve more reliably, more cheaply, and more interpretably.
Related capabilities
More from Models
Ask about this story
"Attention Is All You Need" is the paper that every AI system you've used in the last five years runs on. Eight people at Google wrote it, most of them then left to start the AI companies now disrupting Google. What would you like to explore?