CUDA: The Accidental Infrastructure of the AI Age
NVIDIA built CUDA to sell more GPUs for gaming. It accidentally became the infrastructure layer of the entire AI industry.
In 2006, NVIDIA released CUDA, the Compute Unified Device Architecture, a programming model that allowed developers to write general-purpose code for NVIDIA GPUs. The pitch was to scientists who wanted to run physics simulations faster. The intended customers were petroleum engineers and financial modellers. Nobody at NVIDIA was thinking about neural networks, which were at that point a marginal academic subfield with limited compute requirements and even more limited credibility.
The GPU's advantage for neural network training came from its architecture. GPUs were designed to perform the same simple operation, a multiply-add, on thousands of data points simultaneously, because that is what rendering pixels requires. Neural network training requires exactly the same operation, at enormous scale, across millions of weight updates per second. When Krizhevsky, Sutskever, and Hinton trained AlexNet in 2012 on two GTX 580 GPUs, they cut training time from weeks to days. CUDA was the reason AlexNet was computationally feasible.
NVIDIA spent the next decade building the ecosystem around CUDA. cuDNN (2014) accelerated deep learning primitives (convolutions, pooling, normalisation) so that every major framework ran faster on NVIDIA hardware. TensorFlow (2015), PyTorch (2016), and JAX (2018) all used CUDA as their default compute backend. Each framework that shipped with CUDA support increased the cost of switching away from NVIDIA hardware. By the time AI compute became an industry-defining resource, the CUDA moat was 15 years deep.
The financial consequence was extraordinary. NVIDIA stock traded at approximately $4 per share adjusted in 2016. By 2024, it had reached $900, a 200x return. The H100 GPU, the primary training accelerator for large language models as of 2023, had a list price of $30,000–$40,000, with waitlists extending to months. Microsoft, Google, Amazon, and Meta each spent billions on NVIDIA compute. The semiconductor for playing video games had become the rate-limiting resource for building general intelligence.
The alternative to CUDA (AMD's ROCm, Google's TPUs, custom silicon from Amazon (Trainium), Microsoft (Maia), and Meta) remained incomplete substitutes. Each major framework technically supported non-NVIDIA hardware; most production deployments ran on CUDA. Jensen Huang's 2006 bet on general-purpose parallel computing had produced a monopoly not through market manipulation but through 17 years of compound ecosystem investment. The infrastructure of the AI age was built on a software stack designed to render graphics faster, by a company that had no particular interest in AI until AI had no particular choice but to use CUDA.
What it actually meant
NVIDIA's GPU compute monopoly was not inevitable. It was built on a decade of CUDA ecosystem investment before AI was commercially important. The moat is not the hardware: it is the 17 years of framework integrations, debugging tools, and institutional knowledge that run on top of it. Jensen Huang's bet in 2006 was not on AI. It was on general-purpose parallel computing. AI was a consequence of that infrastructure decision, not its cause.
What enterprises get wrong today
Enterprises model GPU costs as a hardware line item and underestimate the infrastructure complexity. H100 availability, CUDA version compatibility, distributed training overhead, and model serving architecture each add their own failure modes. Teams that successfully run a demo on a single GPU often discover that production deployment at scale requires a fundamentally different architecture.
Related capabilities
More from Moments
Ask about this story
NVIDIA built CUDA to sell GPUs for gaming and scientific computing. In 2012, it accidentally became the reason AlexNet was possible, which started the deep learning revolution, which made NVIDIA briefly the most valuable company in the world. What would you like to explore?