GPT-1: The Quiet Beginning of the Language Era

In June 2018, OpenAI published "Improving Language Understanding by Generative Pre-Training." The paper described training a large language model on a diverse corpus of text, then fine-tuning it on specific downstream tasks. The model had 117 million parameters, large for the time, though GPT-4 would later have an estimated 1.7 trillion. It performed well on a range of NLP benchmarks. The reception was muted. The AI community was focused on BERT, which Google would publish four months later.

The Transformer architecture from the previous year (2017) had opened up a new possibility: you could pre-train on vast amounts of unlabelled text and then adapt the model to specific tasks with relatively small amounts of task-specific data. This was not a new ideatransfer learning had existed in computer vision for years, but the scale and generality were new. GPT-1 demonstrated it worked for language.

What OpenAI's team had noticed was something subtle: the more text you trained on, and the larger the model, the better the pre-trained representations became. This was not an assumption they proved rigorously at first, it was an empirical pattern. GPT-2 (2019, 1.5 billion parameters) extended it. GPT-3 (2020, 175 billion parameters) made it undeniable. The "scaling hypothesis," that capability increases reliably with compute and data, was becoming a research program.

GPT-2 was famous not for its capabilities but for OpenAI's decision not to release its full weights. The model, they said, was too dangerous, it could generate convincing fake news. The decision was widely mocked. GPT-2's outputs, viewed two years later, read as charmingly incoherent. But the public debate it triggered about responsible release was the first time AI capability and AI risk entered the same conversation in mainstream culture.

ChatGPT launched in November 2022, built on GPT-3.5. It reached 100 million users in 60 days. ELIZA, the first chatbot, had launched in 1966 with a pattern-matching script and disturbed its creator with how readily people confided in it. Fifty-six years later, GPT-1's descendants were disturbing everyone. GPT-1 itself is barely remembered. The footnote that started it all.