Random Forests: Wisdom of the Crowd, One Tree at a Time
Random Forests exist because one scientist was frustrated by noisy single trees.
Leo Breiman had a problem with decision trees. They were brilliant individually, interpretable, fast, and surprisingly accurate, but fragile. Change a few training examples, and the tree changed radically. The splits were unstable. The predictions were high-variance. Decision trees, it turned out, were like asking one person for directions: often useful, occasionally disastrously wrong.
Breiman's intuition was that you shouldn't ask one tree. You should ask many trees, each trained on a slightly different sample of the data, and let them vote. In 1996, he formalised this into "Bagging," bootstrap aggregating, where you sample the training data with replacement, train a model on each sample, and average the predictions. The ensemble was more stable than any individual model. Bagging worked, but it had a flaw: the trees were still correlated. If one strong predictor dominated, all the trees looked similar.
The final insight came in 2001. Breiman's Random Forests paper added a second layer of randomness: when building each tree, instead of considering all features at each split, consider a random subset. Force each tree to be different, not just trained on different data. Now your trees were genuinely independent. The ensemble's variance dropped further. Predictions stabilised. And accuracy, on many benchmarks, surpassed everything available, including SVMs, which had dominated the previous decade.
"Random Forests" was published in the journal Machine Learning in 2001, when Breiman was 67 years old. He had already had a distinguished career in statistics, but the Random Forest paper became his most cited work by a wide margin. It is now standard in clinical trials, credit scoring, ecological modelling, and any domain where interpretability and robustness matter alongside accuracy.
Breiman died in 2005, four years after the paper. Random Forests are still widely used today, often outperforming more complex methods on tabular data, where they remain a first-choice baseline for data scientists who want something that works without hyperparameter heroics. The lesson: sometimes frustration with a single noisy expert is the beginning of something that scales.
What it actually meant
Breiman's insight was that ensemble diversity requires structural randomness, not just data variation. Correlated trees make correlated errors; decorrelated trees make independent errors that cancel each other out. Building things that disagree on principle, not just by chance, is a design decision.
What enterprises get wrong today
Data scientists reach for deep learning on tabular enterprise data because neural networks feel modern. Random Forests and gradient-boosted trees routinely outperform deep networks on structured data like transactions, logs, and forms, where the inductive biases of deep learning provide no advantage. The wrong model class wastes compute and reduces accuracy.
Related capabilities
More from Algorithms
Ask about this story
Leo Breiman was frustrated by how brittle individual decision trees were, so he asked many trees, made each one a little different, and let them vote. It's one of the simplest ideas in ML that still works better than methods ten times more complex. What would you like to know?