MMLU: The Benchmark That Measures What a Model Actually Knows

In 2021, Dan Hendrycks and colleagues at UC Berkeley published "Measuring Massive Multitask Language Understanding." The paper's premise was that existing NLP benchmarks measured narrow capabilities (reading comprehension of one type, sentiment of another) but had no way to assess the breadth of knowledge a language model had absorbed. They assembled 57 subjects spanning elementary mathematics, US history, computer science, law, medicine, physics, moral philosophy, and professional accountancy. Questions came from actual exams: the USMLE medical licensing exam, the Bar, standardised tests, and academic assessments. The full benchmark contained 15,908 multiple choice questions. They called it MMLU.

The benchmark arrived at precisely the right moment. GPT-3 had been released in 2020, and the field was trying to understand what very large language models actually knew. MMLU provided a structured answer. GPT-3 scored around 43% on MMLU: well above random chance for four-way multiple choice (which would be 25%), but strikingly low for a model that could write fluent essays. This was surprising to some and clarifying to others. A model that could compose legal briefs could not reliably pass an elementary school history test. The gap between fluency and knowledge was now measurable.

As models improved, MMLU became the primary comparison axis. GPT-4 scored 86.4%. Claude 3 Opus scored 86.8%. Gemini Ultra scored 90.0%. Llama 3 scored 79.5%. Every major model release included an MMLU score, and the number functioned as a credibility signal. Investors, researchers, and journalists used MMLU performance as shorthand for general capability. The benchmark was imperfect (critics noted that models could score well by pattern-matching exam formats rather than demonstrating genuine understanding) but no alternative achieved the same consensus.

MMLU's most important contribution was establishing multitask breadth as a dimension of model quality distinct from single-task performance. Before MMLU, a model could be state-of-the-art on SQuAD without knowing anything about medicine or law. MMLU forced the field to ask whether models were generally intelligent or narrowly competent. It became the benchmark that separates general-purpose models from specialised ones, and its scope (57 subjects, professional to elementary) remains the best available proxy for the question nobody knows how to answer directly: how much does this model actually know?