SQuAD: The Reading Comprehension Exam That Defined NLP for a Decade

In 2016, Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang at Stanford published SQuAD, the Stanford Question Answering Dataset. The construction method was elegant: they selected 500 Wikipedia articles, hired crowdworkers on Mechanical Turk to read each passage and write questions about it, and then asked a second set of workers to identify the exact span of text in the passage that answered each question. The result was 107,785 question-answer pairs where every answer was a contiguous substring of the source passage. You could not generate answers; you could only find them.

SQuAD defined machine reading comprehension as a concrete task. Before it, reading comprehension was a vague capability attributed to systems in qualitative terms. SQuAD made it measurable: given a passage and a question, extract the correct answer span. The evaluation metric was F1 score over character-level overlap. Human performance on the benchmark was measured at 82.3% F1 (humans sometimes disagree on the exact span). Every model published between 2016 and 2018 reported its SQuAD score alongside its accuracy on other benchmarks.

The benchmark became a competition. IBM, Google, Microsoft, and Allen AI all built systems targeting SQuAD. Models improved from roughly 60% F1 in 2016 to 75% by mid-2017. Then in October 2018, Google released BERT. Its SQuAD score was 93.2% F1: surpassing the human baseline of 82.3%. For the first time, a machine had exceeded human performance on a reading comprehension benchmark. The NLP community's reaction was a mixture of celebration and unease. If the model scored higher than humans, did it understand the passages? Or had SQuAD been solved by pattern matching?

The question led to SQuAD 2.0, published in 2018, which added 53,775 unanswerable questions: questions that looked answerable but had no correct span in the passage. Systems were required to know when not to answer. This was significantly harder. The best systems dropped to around 66% F1 before rebounding to near-human performance again over the following two years. SQuAD and SQuAD 2.0 together defined the frontier of NLP capability for nearly five years, shaped the architecture of BERT, and established reading comprehension as the central metaphor for what language understanding means.