GSM8K: The Grade-School Problems That Unlocked Chain-of-Thought

In 2021, Karl Cobbe and colleagues at OpenAI published "Training Verifiers to Solve Math Word Problems" and released GSM8K, Grade School Math 8K. The dataset contained 8,500 grade-school math problems requiring two to eight steps of arithmetic reasoning. Problems like: "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did she sell altogether?" Every problem was solvable with elementary arithmetic; none required algebra, calculus, or memorised formulas. They were the kind of problems an average ten-year-old could solve with pencil and paper.

GPT-3 scored approximately 35% on GSM8K with standard prompting: less than half. This was striking. A model that could write legal briefs, translate poetry, and generate working code was failing at problems designed for children. The failure revealed something specific about how large language models processed reasoning: they could retrieve facts and generate plausible text, but they struggled to execute multi-step arithmetic procedures where each step depended on the output of the previous one.

The discovery that unlocked GSM8K came in a 2022 Google Brain paper by Jason Wei and colleagues: chain-of-thought prompting. Instead of asking the model to output an answer directly, the prompt included examples where reasoning steps were written out explicitly. "Natalia sold 48 clips in April. She sold half as many in May, which is 48/2 = 24 clips. The total is 48 + 24 = 72 clips. The answer is 72." When models were given examples in this format, their accuracy on GSM8K jumped dramatically. GPT-4 eventually reached above 90%. The benchmark did not change. The prompting did.

GSM8K became the canonical benchmark for mathematical reasoning in language models. Every reasoning paper published after 2022 reports GSM8K accuracy. Every new model announcement includes it. It drove the development of process reward models, which verify each reasoning step rather than only the final answer, and self-consistency, which samples multiple reasoning chains and takes the majority answer. The problems are elementary but the implications are deep: GSM8K is the benchmark that proved that how you ask matters as much as what you ask, and that explicit reasoning, written out step by step, dramatically changes what a model can do.