HumanEval: The Benchmark That Measures Whether AI Can Actually Code

In 2021, Mark Chen and colleagues at OpenAI published "Evaluating Large Language Models Trained on Code" and introduced Codex, the model that would become GitHub Copilot. To evaluate it, they needed a benchmark that measured functional correctness rather than surface similarity. Existing code benchmarks compared generated code to reference solutions using string matching or token overlap, which rewarded stylistic similarity over functional equivalence. A correct solution that used a different variable name would score poorly. A wrong solution that used the right variable names would score well. This was useless.

Chen's team hand-wrote 164 Python programming problems, each consisting of a function signature, a docstring describing what the function should do, and a set of unit tests that verified correct behaviour. The evaluation metric was simple: run the generated code against the unit tests. Pass all tests, score one point. Fail any test, score zero. They measured pass@k: the probability that at least one of k generated samples passes the tests. Pass@1 is the probability that a single attempt succeeds. This evaluation methodology (run the code and check whether it works) became the standard for all subsequent code generation benchmarks.

HumanEval was a narrow but honest benchmark. The 164 problems were tractable but not trivial: they included string manipulation, mathematical operations, list processing, and simple algorithmic tasks. Codex achieved 28.8% pass@1. GPT-4 later achieved around 67%. Claude 3.5 Sonnet reached above 90%. The progression across three years of model development is visible in a single number, making HumanEval one of the cleaner records of capability improvement in AI history.

The benchmark also hid a trap that took the field years to reckon with: passing every unit test is not the same as writing good code. Early audits of Codex completions found solutions that hardcoded the exact values a test expected, introduced subtle bugs the narrow tests never exercised, or solved the stated problem in a way too brittle to survive a changed specification. A 90% pass@1 score said a model could satisfy a short, self-contained function with a clear docstring. It said nothing about whether that model could work inside an existing fifty-thousand-line service, respect its conventions, or avoid breaking a caller three files away: exactly the gap that agentic benchmarks like SWE-bench were later built to expose.

The benchmark spawned an ecosystem. MBPP (Mostly Basic Python Problems), EvalPlus (a harder version with additional test cases), HumanEval+, and APPS (for more complex algorithm problems) all built on HumanEval's methodology. The pass@k metric became the standard evaluation approach for code generation across languages and frameworks. HumanEval itself was criticised for data contamination (the problems may have appeared in training data) but its methodology proved more durable than any individual result. The insight that code correctness should be evaluated by running code, not by reading it, was the enduring contribution.

By the mid-2020s, frontier models were clearing HumanEval at rates that made the benchmark nearly uninformative on its own, scores clustered near the ceiling, and the real differentiation moved to harder successors: LiveCodeBench, which refreshes its problem set on a rolling basis to fight contamination, SWE-bench, which scores models against real GitHub issues in real repositories, and agentic coding evaluations that measure whether a model can plan, edit, run, and debug across a multi-file project rather than fill in a single function. HumanEval did not disappear so much as get absorbed: it is still reported in nearly every model card released in 2026, now as a baseline sanity check rather than a frontier, the same way MNIST still turns up in computer vision papers long after the field moved on.