The Breakthrough Behind Modern AI Was Mostly About Being Able to Use the Whole GPU

The Breakthrough Behind Modern AI Was Mostly About Being Able to Use the Whole GPU

Attention is usually explained as the mechanism that lets a model "focus on the important words," which is true and mostly unhelpful. It makes the innovation sound like a cognitive metaphor when the actual reason it took over is closer to an engineering one.

Here is the version that explains why everything changed after 2017.

The Problem With Reading in Order

Before transformers, sequences were processed by recurrent networks — RNNs, and their better-behaved descendants LSTMs and GRUs. These read one token at a time. Token one, then token two, then token three, each step producing a hidden state passed forward to the next. A game of telephone with a fixed-size note.

That design has two problems, and the second one is the one that mattered.

Information degrades over distance. If token 50 needs something from token 3, that information has to survive 47 intermediate steps, being compressed, mixed with new input, and partially overwritten at each one. LSTMs added gates to help important information persist, which helped, but the structural bottleneck remained: everything had to flow through every intermediate position. Consider a sentence where the subject at the start determines the verb form much later; by the time the network reaches the verb, the subject details have been diluted across dozens of updates, often beyond recovery. The same pattern appears in code completion or long-document summarization, where early context is essential but arrives too degraded to be useful.

You cannot parallelise it. This is the fatal one. Token 5 cannot be processed until tokens 1 through 4 are done, by construction. On hardware whose entire performance advantage comes from doing thousands of things simultaneously, a design that insists on doing one thing at a time is leaving nearly all the capability unused. Modern GPUs contain thousands of cores and specialized tensor units optimized for bulk matrix operations; a sequential loop touches only a single core at each step and leaves the rest idle. Training runs that should have taken days stretched into weeks or months simply because the architecture could not feed the hardware enough work at once.

That second constraint is what capped the scale of what could be trained. Not ideas, not data — the fact that the architecture could not exploit the machine.

What Attention Actually Does

Consider the sentence the bank approved the loan.

After the earlier stages of the pipeline, you have five vectors, one per token, each carrying some meaning and some position information. But they are isolated — each sits in its own row, unaware of the others. The vector for bank does not yet know that loan appears three positions later, which is the entire basis for concluding this is a financial institution rather than a riverbank. The same isolation occurs across entire paragraphs: a pronoun in one sentence may refer to a noun several sentences earlier, yet nothing in the representation links them until later processing.

Attention connects them. Each token compares itself against every other token, produces a set of relevance weights, and builds a new representation of itself as a weighted blend of everything it found relevant.

Bank looks at all five positions, weights loan and approved heavily, weights the near zero, and updates itself accordingly. It is now a representation of "bank, in a financial context," derived from evidence rather than assumption. The comparison is done with dot products that measure how aligned two vectors are; higher alignment produces a larger weight, so the model effectively votes on which other tokens matter for its own updated meaning.

The distance between the two words is irrelevant. Three positions or three hundred — it is one comparison either way, not a chain of hops. That solves the first problem completely.

Why It Solved the Second Problem Too

Every one of those comparisons is independent. Bank comparing itself to loan does not depend on approved having finished anything.

Which means the whole operation is a matrix multiplication. Every token against every token, all at once, in one operation, on hardware built to do exactly that as fast as physically possible. The input is packed into a single matrix, the comparisons become one batched multiply, and the weighted sums that follow are another. Modern GPU kernels are written precisely for this pattern; they stream the data through thousands of cores in parallel rather than stepping through it sequentially.

The sequential dependency is gone. Training can now use the full width of the machine across the entire sequence, and it is that — more than any conceptual leap — that made it practical to train models orders of magnitude larger than what came before.

The mechanism is elegant, but the reason it won is that it fit the hardware. That is a recurring pattern in computing history and it is worth noticing: the winning design is frequently not the cleverest one but the one whose work can be spread across the available silicon. Earlier successes such as convolutional networks in vision followed the same logic; they replaced sequential feature extraction with parallel filter applications that GPUs could execute in bulk.

The Cost, Which Is Real

Comparing every token to every other token means the work grows with the square of the sequence length. Double the context and you quadruple the comparisons.

This is why context windows were a hard constraint for years, and why so much subsequent research went into ways of getting attention's benefits without paying its full quadratic price. Early transformer models were limited to a few hundred or a couple thousand tokens before memory use became prohibitive; researchers responded with techniques such as local windows, sparse patterns, and linearized approximations that reduce the cost while keeping most of the connectivity. The original design traded memory and compute for parallelism, and that trade was overwhelmingly correct — but it is a trade, and the bill arrives as sequences get long.

What This Is Actually Built From

The thing worth taking away, if you are the sort of person who suspects there is more magic here than there is: the components are all ordinary.

Tokenization is splitting text into pieces and mapping them to integers. Embeddings are lookup tables of learned vectors. Attention is dot products and a softmax. The transformer block is attention plus a small feed-forward network plus normalisation, repeated. Training is comparing predictions to actual next tokens, measuring the error, and adjusting weights to reduce it.

None of these is beyond an undergraduate maths course. A working small-scale version of the whole architecture is a few hundred lines of code, runs on a normal machine, and produces recognisable text. The individual operations map directly onto standard linear algebra routines that have been optimized for decades.

What makes the frontier models frontier models is scale — data, parameters, compute, and an enormous amount of engineering in getting training to converge at that scale. The architecture underneath is comprehensible, and the gap between understanding it and being able to build one is a matter of resources rather than a matter of secrets.

That is worth knowing in both directions. It should make the technology less mystifying, and it should make you more sceptical of anyone describing it in language that implies nobody knows how it works.

The Fundamentals of Training an LLM: A Python & PyTorch Guide builds the whole thing from first principles — loss functions, gradient descent, tokenization, embeddings, attention, the transformer block, a complete working model, training it, fine-tuning with LoRA, and what changes at scale.