AG News: The Benchmark That Proved Text Classification Could Scale

In 2015, Xiang Zhang, Junbo Zhao, and Yann LeCun published "Character-level Convolutional Networks for Text Classification" at NeurIPS. The paper asked a simple question: if you treat text as a sequence of characters rather than words, can a convolutional network still classify it accurately? To answer it, they assembled several datasets, the most enduring of which was AG News, a collection of news articles from AG's Corpus, a dataset of over one million news articles gathered from more than 2,000 news sources. They took 120,000 training articles and 7,600 test articles across four categories: World, Sports, Business, and Science and Technology.

The appeal of AG News as a benchmark was its combination of scale, difficulty, and speed. At 120,000 training examples it was large enough to train neural models without overfitting immediately. The four-way classification was harder than binary sentiment but still tractable. And news articles, unlike tweets or reviews, had consistent length and register, making the dataset's variance lower than noisier text sources. Models trained on AG News generalised predictably, which made it reliable for comparing architectures.

Zhang and LeCun's paper demonstrated that character-level models could match or exceed word-level models on text classification, which was genuinely surprising. The dominant assumption was that words were the natural unit of meaning in text. Characters seemed too low-level. The paper showed that with sufficient depth and data, the network could learn word-level and phrase-level structure from character sequences alone. This finding had implications beyond classification: it suggested that linguistic structure could be learned rather than engineered, reinforcing the pattern that would culminate in large language models.

AG News became the standard benchmark for text classification speed. FastText, Facebook AI's 2016 library for efficient text classification, reported AG News accuracy as its primary comparison point. Papers on distillation, pruning, and quantisation for NLP almost uniformly include AG News in their benchmark suites because its training set is large enough to be meaningful and its evaluation is fast enough to run many times. It sits at the intersection of academic research and industrial deployment: large enough to matter, small enough to iterate on quickly.