Word2Vec: The Geometry of Language

In January 2013, Tomas Mikolov and colleagues at Google published a paper that demonstrated something that felt like magic: you could represent words as vectors in a high-dimensional space, and the arithmetic of those vectors corresponded to the semantics of language. King minus Man plus Woman equalled something close to Queen. Paris minus France plus Italy equalled something close to Rome. The geometry of meaning could be computed.

The idea built on decades of distributional semantics, the hypothesis that words appearing in similar contexts have similar meanings. Mikolov's insight was computational efficiency. Previous models that captured word context required expensive matrix factorisation. Word2Vec used a simple shallow neural network, either predicting a word from its context (CBOW) or predicting context from a word (Skip-gram), trained on billions of tokens. The training was fast. The embeddings were small. And they worked remarkably well.

What made Word2Vec's embeddings strange and wonderful was that they captured relational structure without anyone programming it in. The model had never been told what "capital city" means. It learned that the relationship between Paris and France was the same kind of thing as the relationship between Rome and Italy, purely from co-occurrence patterns in text. Analogy was an emergent property of language statistics.

Word2Vec became the standard pre-processing step for virtually every NLP task through the mid-2010s. Before you classified sentiment, parsed syntax, or extracted named entities, you embedded your words. The technique spawned Doc2Vec (for paragraphs), Node2Vec (for graphs), and eventually contributed to the intuition that led to contextualised embeddings like ELMo and BERT.

In 2014, Mikolov left Google for Facebook AI Research. He did not patent Word2Vec or keep the weights proprietary. He released the code and the pre-trained vectors openly. This decision, to share rather than protect, allowed Word2Vec to propagate through academia and industry simultaneously, seeding the culture of open pre-trained models that defines AI research today. The geometry of meaning, it turned out, was something you could give away.