1996Datasets & Benchmarks
Adult Income: The Dataset That Exposed ML's Fairness Problem
The Adult Income dataset was extracted from a census to study poverty, and accidentally became the standard test for algorithmic fairness.
Datasets & Benchmarks
Datasets & Benchmarks in AI History
The datasets and benchmarks that defined what problems AI would solve - and how we measure progress.
14 stories
1997Datasets & Benchmarks
California Housing: The Benchmark That Replaced a Biased One
California Housing replaced the Boston dataset after researchers found Boston embedded racial data in a feature called "lower status." California became the clean regression standard.
2009Datasets & Benchmarks
CIFAR-10: The Benchmark Between MNIST and ImageNet
CIFAR-10 was assembled by Alex Krizhevsky in Hinton's lab, and the techniques he developed to crack it became AlexNet.
2011Datasets & Benchmarks
IMDB Sentiment: The Dataset That Brought NLP into the Deep Learning Era
Andrew Maas scraped 50,000 IMDB reviews to study word vectors for sentiment, and created NLP's first large-scale transfer learning benchmark.
2012Datasets & Benchmarks
Titanic: The Dataset That Taught a Million People Data Science
Kaggle launched with the Titanic dataset and accidentally created the on-ramp for a generation of data scientists.
2013Datasets & Benchmarks
Credit Card Fraud: The Dataset That Taught ML to Handle Imbalance
A European credit card dataset with 492 frauds in 284,807 transactions taught the field how to handle extreme class imbalance.
2015Datasets & Benchmarks
AG News: The Benchmark That Proved Text Classification Could Scale
AG News was assembled to test whether character-level convolutions could classify text, and it became the standard benchmark for fast, scalable text classification.
2016Datasets & Benchmarks
SQuAD: The Reading Comprehension Exam That Defined NLP for a Decade
SQuAD turned Wikipedia into a reading comprehension exam, and BERT passed it at human level in 2018, a moment that stopped the field.
2019Datasets & Benchmarks
ARC-AGI: The Benchmark Built to Resist Everything AI Was Good At
François Chollet designed ARC-AGI to be unsolvable by memorisation. It required visual pattern generalization from just a few examples. For five years, top AI systems scored near zero.
2021Datasets & Benchmarks
MMLU: The Benchmark That Measures What a Model Actually Knows
MMLU asked language models 57 different exam questions, from elementary school to medical licensing, and became the single number every LLM is judged by.
2021Datasets & Benchmarks
GSM8K: The Grade-School Problems That Unlocked Chain-of-Thought
GSM8K was 8,500 grade-school math problems, and GPT-3 failed at them. Then chain-of-thought prompting was discovered, and they became solvable.
2021Datasets & Benchmarks
HumanEval: The Benchmark That Measures Whether AI Can Actually Code
OpenAI built HumanEval to measure Codex: 164 hand-written Python problems that became the standard exam for every code-generating AI since.
2023Datasets & Benchmarks
GAIA: The Benchmark That Showed How Far Agents Have to Go
GAIA asked AI assistants 466 real-world questions that a human with internet access could answer in minutes. Humans scored 92%. GPT-4 with tools scored 15%.
2024Datasets & Benchmarks
SWE-Bench: The Benchmark That Measures Real Software Engineering
SWE-Bench pulled 2,294 real GitHub issues from production codebases. Early models solved 2% of them. Then AI coding agents became the fastest-moving benchmark in history.