How a GPT-style Transformer Works
A GPT-style transformer does two jobs over and over: it lets tokens share context through attention, then gives each token a small neural network to work on what it learned. Most of the architecture helps those two updates happen reliably, many layers deep.
Language-model architecture
Take the sentence “The trophy didn’t fit in the suitcase because it was too small.” To make sense of “it,” you need something from earlier in the sentence. At the point where the model reads “it,” it can use “trophy” and “suitcase” but hasn’t seen “small” yet. Once it sees “small,” that new position can use the whole prefix to make the next prediction.
That restriction is the starting point for a GPT-style transformer. Every position gets a vector. Attention lets it bring in information from earlier positions, and a small neural network at each position then works with the result. Run that pair of operations several times and use the final vector to score possible next tokens.
From text to a row of vectors
A tokenizer splits the text into pieces and replaces each with an integer ID. A piece might be a word, part of a word, or punctuation. The model looks up a learned vector for each ID. In GPT-2 it also adds a learned vector for each position, so the same token at position 3 starts with a different representation than it does at position 20.
With five input tokens and a model width of 768, there are five rows of 768 numbers. The block takes those five rows in and returns five rows of the same width. What changes is what those rows contain.
One attention head with three tokens
Each row is used to make three vectors called a query, a key, and a value. For a given position, the query is compared with the keys using dot products. Those scores are scaled, future positions are masked out, and softmax turns the scores into weights that add up to one. The head uses those weights to sum the value vectors.
Here is a made-up head looking at the last position of “the cat slept.” To keep the arithmetic short, the example uses scalar values and starts after the softmax. The real model learns the projections for all three vectors.
Tokens: the cat slept
Attention weights: 0.10 0.70 0.20
Toy values: 1 3 2
Head output = (0.10 × 1) + (0.70 × 3) + (0.20 × 2) = 2.6What happens after the tokens talk
Several heads do that calculation in parallel, each with its own projections. Their outputs are joined and projected back into the model width. A residual connection adds the result to the original rows so attention can update a representation without replacing it outright.
Next comes the MLP, a small feed-forward network that applies the same weights independently to every row. GPT-2 expands each row to four times its width, applies GELU, and projects it back before adding another residual connection. Layer normalization precedes each of these two updates in GPT-2. Once the block finishes, the next block starts with the updated rows.
Why training gets more parallel work than generation
For a training sequence like “the cat slept,” the model can use the row for “the” to predict “cat” and the row for “cat” to predict “slept” in the same forward pass. The causal mask prevents either row from peeking at its answer. The output projection produces a score for every vocabulary item, and cross-entropy penalizes the model when it gives the actual next token a low probability.
When generating, the next token is unknown. The model scores it from the final position, selects a token, and appends it. A key-value cache saves the earlier attention projections, which avoids recalculating them for every new token. Each new token still has to wait for the previous one, and the cache grows as the sequence gets longer.