What was actually broken in AI before transformers?

You have probably heard some version of the same origin story. Neural networks pottered around for decades doing basically nothing, and then one day, boom, transformers showed up and the lights turned on. It makes for a nice narrative. It is also not really true.

By 2012, image recognition networks were already crushing benchmarks that had stumped researchers for years. Speech recognition and machine translation were already working reasonably well too, powered by something called recurrent neural networks, RNNs, and a smarter cousin called LSTM. These things were not toys. They shipped in real products.

So hold on, if neural networks already worked, what was actually missing?

The problem was not that these networks were dumb. The problem was how they read. RNNs process a sentence one word at a time, left to right, the way you might read while covering everything except the current word with your hand. Word ten cannot start until word nine has finished.

Two things fall out of reading like that. First, it is painfully slow, because you cannot work on several words at once, everything sits in a queue. Second, and this is the one that actually mattered, memory fades. There is a real mathematical effect called the vanishing gradient problem, where information from early words gets diluted so much that by the time the model reaches word fifty, it has practically forgotten word two ever existed.

So by the mid 2010s there was a very specific, well understood bottleneck. Not "AI does not work." More like, "AI forgets what you said thirty seconds ago, and there is nothing anyone can do about it, mathematically." That is the itch a very specific idea from 2017 was about to scratch.