PCA: The Algorithm That Finds the Shape Hidden in Your Data

In 1901, Karl Pearson published "On Lines and Planes of Closest Fit to Systems of Points in Space" in the Philosophical Magazine. His stated goal was modest: given a cloud of points in multi-dimensional space, find the line or plane that minimised the sum of squared perpendicular distances to all points. The technique he developed was principal component analysis: finding the direction of maximum variance in the data, then the direction of maximum remaining variance orthogonal to the first, and so on. He called these the principal axes. We now call them principal components.

Harold Hotelling independently reinvented PCA in 1933, framing it in terms of psychological measurement. He was trying to extract latent variables from test scores: if a student's performance across many different tests is correlated, what underlying traits explain that correlation? His formulation introduced the modern algebraic description: the principal components are the eigenvectors of the covariance matrix, ordered by their corresponding eigenvalues. This formulation made PCA analytically tractable and connected it to linear algebra in a way that Pearson's geometric description had not.

PCA became foundational wherever high-dimensional data appeared before machine learning existed to process it. Chemometrics used it to analyse spectroscopic measurements in the 1970s. Genetics used it to reveal population structure in genome-wide association studies: plotting the first two principal components of genetic data across thousands of people organises them into approximate geographic clusters without any prior labels. Econometrics used it to reduce correlated financial variables into independent factors. In each domain, PCA answered the same question: what are the real dimensions of this data?

Computer vision gave PCA its most famous application. Sirovich and Kirby (1987) and Turk and Pentland (1991) demonstrated eigenfaces, the principal components of a collection of face images. Each eigenface is a ghostly composite capturing common facial variation patterns. Any face can be represented as a linear combination of eigenfaces, dramatically reducing the number of numbers needed to store or compare faces. This was the dominant face recognition approach through the 1990s and early 2000s, before deep learning replaced it.

In modern machine learning, PCA is the default first step for exploratory data analysis on high-dimensional datasets: project the data into two or three dimensions and look at it before modelling. t-SNE and UMAP have largely replaced PCA for visualisation because they preserve local structure better. But PCA's interpretability advantage remains: each component has a clear meaning as a direction of maximum variance, and the percentage of variance explained by each component is immediately informative. Pearson's 1901 geometry paper still runs daily in genomics labs, finance teams, and ML preprocessing pipelines worldwide.