Why does the whole AI boom run on chips built for video games?

Quick one that trips people up constantly. Why does none of this run on a regular computer chip?

A CPU is a small team of geniuses. Four cores, eight, maybe sixty four on a serious server, and each one is very good at handling complicated logic, branching decisions, one task after another in sequence.

A GPU is the opposite philosophy entirely. Thousands of cores, sometimes more than ten thousand, and each one is honestly kind of dumb on its own. It cannot juggle complex decisions. What it can do is take one tiny, simple operation, multiply these two numbers and add the result, and run it across an enormous pile of data all at the same time. An army of simple workers instead of a handful of brilliant ones.

Ok but why would that matter for something as abstract as language?

Because that self attention comparison, every word checked against every other word, is under the hood just thousands of tiny identical multiplications that do not depend on each other. Hand that to a CPU and it grinds through them mostly one at a time. Hand it to a GPU and it assigns each one to a different core and fires them all off together.

And here is the part that makes this genuinely funny. GPUs were not built with any of this in mind. GPU literally stands for graphics processing unit. They were built to render video games. A screen is a grid of millions of pixels, and thirty or sixty times a second you need to work out the colour of every single one. Calculating pixel one has nothing to do with calculating pixel two, they are all tiny, independent, identical little jobs.

So through the nineties and two thousands, chip makers, Nvidia especially, got extremely good at exactly that shape of problem, purely to make games look better. Then around 2012, researchers working on image recognition noticed something. Training a neural network is mathematically the same shape of problem. Huge grids of numbers, matrix multiplication, embarrassingly parallel. So they hijacked gaming chips for it.

A chip built to draw explosions in a shooter game turned out to be exactly what language needed.

Nvidia noticed researchers doing this and leaned in hard, building a software layer called CUDA specifically so people could program these chips for general maths instead of only graphics. That single bet is a huge part of why Nvidia now sits among the most valuable companies on the planet, off the back of a market nobody designed the chip for in the first place.