ARC-AGI: The Benchmark Built to Resist Everything AI Was Good At

In 2019, François Chollet, the creator of Keras, published "On the Measure of Intelligence," a 64-page paper arguing that the field had been measuring the wrong things. Accuracy on fixed benchmarks, he argued, measured skill at a specific task, not intelligence. A system trained on millions of examples could achieve high benchmark scores through memorisation and interpolation, not through genuine generalisation. To measure generalisation, you needed tasks that could not be solved by pattern-matching on training data. The Abstraction and Reasoning Corpus, ARC, was his attempt to build such a task.

ARC presented systems with visual grids of coloured squares. Each puzzle contained three to five demonstration pairs showing an input grid and its corresponding output grid. The system was asked to produce the output for a new input grid, applying whatever transformation rule it had inferred from the demonstrations. The rules were abstract: reflections, rotations, filling enclosed regions, extending patterns, counting objects. A seven-year-old could usually solve them given a minute of thought. The rules were never stated; they had to be inferred from just a few examples. No rule appeared in more than a handful of puzzles, making memorisation useless.

For five years after publication, top AI systems scored near zero on ARC. GPT-4 in its standard form achieved around 0-5%. Systems trained specifically on ARC-style puzzles improved somewhat but did not generalise. The benchmark appeared to have succeeded in measuring something that contemporary AI genuinely could not do. ARC Prize, a competition launched in 2024 with a $1 million prize, brought renewed attention. Then in December 2024, OpenAI's o3 model achieved 87.5% on ARC-AGI in high-compute mode, stunning the research community and prompting Chollet to argue that ARC had not been solved but that the goalposts needed to move.

ARC-AGI's legacy is its clarity of purpose. Unlike most benchmarks, which measure performance on tasks humans designed for other reasons, ARC was designed from first principles to measure a specific cognitive capability: few-shot abstract pattern generalisation. Its five-year resistance to state-of-the-art AI systems made it the most credible measure of the gap between current AI and human-level general reasoning. Its eventual partial solution by o3 raised more questions than it answered: whether the 87.5% came from genuine generalisation or from massive compute searching over solution strategies. The debate about what ARC-AGI actually measures is, appropriately, a debate about what intelligence actually is.