Mixture of Experts: The Forgotten Architecture Behind the Frontier

In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton published "Adaptive Mixtures of Local Experts" in Neural Computation. The idea was straightforward: instead of one network learning everything, train multiple specialised networks and a gating network that routes each input to the right expert. Different experts would specialise on different regions of the input distribution. The ensemble would be more efficient than a single large network because each expert only modelled a subset of the problem. It was an elegant idea that arrived before the field had the scale to use it.

For three decades, Mixture of Experts remained theoretically interesting but practically underused. The key innovation for large-scale use was sparsity: activate only a subset of experts per input token rather than all of them. This meant you could scale model capacity (total parameters) without proportionally scaling inference compute (active parameters per forward pass). A model with 8 experts activating 2 per token would have the capacity of a model 4 times its computational cost. Capacity and compute were decoupled for the first time.

Google Brain published "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" in 2017, demonstrating the technique at scale in language models. Fedus, Zoph, and Shazeer's Switch Transformer (2021) scaled it to 1.6 trillion parameters with only 2 billion active per token. The load-balancing problem (ensuring tokens were routed evenly across experts rather than collapsing onto a few) required an auxiliary loss to avoid expert collapse. The technique worked at scale but was finicky to train.

In December 2023, Mistral AI released Mixtral 8x7B, an open-source MoE model with 8 experts of 7 billion parameters each, activating 2 per token (12.9 billion active parameters per inference). It matched GPT-3.5 on most benchmarks while running faster than a dense 13B model. The open release demonstrated that MoE was not just a technique for frontier labs: the architecture could produce frontier-level quality at mid-tier inference cost. Within weeks, Mixtral 8x7B was the default recommendation for teams needing near-GPT-4 quality at GPT-3.5 price.

The most significant deployment of MoE is reportedly GPT-4, which leaked descriptions suggest is composed of 8 expert models of approximately 220 billion parameters each. If accurate, GPT-4 had over 1.7 trillion total parameters but activated roughly 280 billion per token. The combination of Jordan and Jacobs' 1991 routing idea with sparse gating had produced the architecture behind the most capable publicly deployed model. A forgotten paper from before the internet had become the skeleton of modern frontier AI.