Batch Normalisation: The Trick That Nobody Fully Understands
Batch Normalisation let researchers train networks 14× faster, and nobody fully understands exactly why it works.
In 2015, Sergey Ioffe and Christian Szegedy at Google published "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." The paper proposed a simple operation: at each layer of a neural network, normalise the activations of each mini-batch to have zero mean and unit variance, then apply learned scale and shift parameters. The claimed benefit was reducing "internal covariate shift," the way the distribution of inputs to each layer changes as the weights in earlier layers update.
The practical results were dramatic. Networks trained with batch normalisation converged 14× faster in their experiments. They were less sensitive to the choice of learning rate. They required less careful weight initialisation. They acted as a mild regulariser, reducing the need for dropout in some architectures. The Inception architecture that swept the 2015 ImageNet challenge used batch norm throughout.
The theoretical explanation, internal covariate shift, has since been questioned. A 2018 paper by Santurkar et al. showed experimentally that batch normalisation does not actually reduce internal covariate shift in the way Ioffe and Szegedy claimed. What it does, they argued, is smooth the optimisation landscape, making the loss function less jagged and allowing larger learning rate steps without divergence. But this explanation is also contested. The field's honest position is: batch normalisation works remarkably well, is used in almost every modern deep network, and is not fully theoretically understood.
Batch norm became so universal that it's almost invisible, a standard component assumed to be present. Its successors (layer normalisation, group normalisation, RMS normalisation) address its limitations in recurrent networks, small batch sizes, and distributed training. But the core principle, normalise activations to control their distribution, remains everywhere. Sometimes the most important techniques are the ones everyone uses and nobody fully understands.
What it actually meant
Batch normalisation works demonstrably and consistently, and the theoretical explanation is genuinely contested. This is more common in ML than researchers like to admit. Empirical evidence is sometimes the whole story available, and shipping something that works is more useful than waiting for a theory that may not arrive.
What enterprises get wrong today
Teams apply batch normalisation uncritically on every layer, in every architecture, at every batch size. Batch norm fails with small batch sizes. It behaves differently during training and inference, creating subtle production bugs when inference batch statistics differ from training. It is not a free improvement. It is a trade-off that requires attention.
Related capabilities
More from Algorithms
Ask about this story
Batch Normalisation made deep networks train 14x faster, and nobody fully agrees on why. The original explanation has been challenged. It just works, remarkably consistently, and the field has moved on. What would you like to explore?