California Housing: The Benchmark That Replaced a Biased One

In 1997, R. Kelley Pace and Ronald Barry published a short paper in Statistics and Probability Letters demonstrating a new approach to spatial autoregression. To illustrate their method, they used a dataset derived from the 1990 US Census: block-level data for California, including median house value, median income, housing age, average rooms, average bedrooms, population, and geographic coordinates. The dataset had 20,640 rows, one per census block group. The paper was narrowly read. The dataset outlasted the paper by decades.

For many years, the default regression benchmark in machine learning was the Boston Housing dataset, 506 observations of Boston neighbourhood statistics collected in 1978 by Harrison and Rubinfeld. Boston was convenient, small, and well-understood. It was also problematic. One of its features, designated B, was constructed as a function of the proportion of residents described as "Black" in the census, based on the authors' assumption that racial composition affected home values. In 2020, the scikit-learn team deprecated the Boston dataset and recommended California Housing as a replacement, noting that Boston's feature construction was ethically unacceptable regardless of its historical context.

California Housing has genuine pedagogical advantages beyond its cleaner provenance. The geographic coordinates (latitude and longitude) make it excellent for teaching spatial visualisation: you can plot house prices across California and see the coast premium, the Bay Area concentration, and the Central Valley gap. The relationship between median income and median house value is strong but nonlinear. The dataset rewards nonlinear models, making it suitable for demonstrating the advantage of gradient boosting over linear regression.

The dataset now ships as a default example in scikit-learn, TensorFlow datasets, and nearly every major ML framework. It is the standard regression benchmark in the same way Titanic is the standard classification benchmark. Its history illustrates something important about how benchmarks work: they are not neutral. The choice of what dataset becomes the default shapes what practitioners think of as normal, what questions they ask, and what problems they consider worth solving. When a benchmark is flawed, retiring it matters.