BERT: Bidirectional Context and the Pre-training Revolution
BERT read sentences backwards and forwards simultaneously, and rewrote how the field thought about language understanding.
In October 2018, four months after GPT-1, Google published BERT, Bidirectional Encoder Representations from Transformers. Where GPT predicted the next token (left-to-right), BERT was pre-trained to predict randomly masked tokens in a sentence, a task that required reading context from both directions simultaneously. The difference was not just architectural. It reflected a different theory of what language understanding requires: not next-word prediction, but filling in context from all available evidence.
BERT's pre-training tasks were deceptively simple. Masked Language Modelling: randomly replace 15% of tokens with a [MASK] token and train the model to predict what was masked. Next Sentence Prediction: given two sentences, predict whether the second follows the first in the original text. These tasks required no labelled data, just text. And the representations learned were startlingly general: fine-tuning BERT on any NLP benchmark required only adding a small classification head and training for a few epochs.
When the BERT paper landed, it set new state-of-the-art results on 11 NLP benchmarks simultaneously. The field had never seen a paper claim improvements across so many tasks at once. Named entity recognition, question answering, sentence classification, natural language inference: BERT outperformed prior methods on all of them, often by large margins. Google integrated BERT into its Search ranking in 2019, describing it as "the biggest leap forward in the past five years."
BERT spawned an entire generation of derivatives: RoBERTa (more data, better masking), ALBERT (parameter reduction), DistilBERT (smaller and faster), domain-specific variants (BioBERT, LegalBERT, FinBERT). The bidirectional pre-training paradigm it established, read everything, learn everything, then fine-tune, became the default for NLP tasks that don't require generation. When people debate GPT vs BERT, they're debating two different theories of language understanding: prediction versus comprehension. Both turned out to be useful.
What it actually meant
BERT's masked language modelling made the model read the whole sentence before predicting any part of it, a training task that forced bidirectional context. The architecture was shaped by the training objective, not the other way around. Task design drives model capability: what you train for determines what you can do.
What enterprises get wrong today
Enterprises fine-tune BERT on small task-specific datasets using hyperparameters documented for large-scale pre-training: wrong learning rate, wrong batch size, no catastrophic forgetting mitigation, no early stopping on domain data. Fine-tuning a pre-trained model on a small dataset requires fundamentally different training dynamics. Teams see poor results and conclude the model does not work for their domain, when the training regime is what is broken.
Related capabilities
More from Models
Ask about this story
BERT improved state-of-the-art on 11 NLP benchmarks simultaneously when it publisheda sweep that had never happened before. And it did it by learning to fill in blanks rather than predict the next word. What would you like to explore?