How Transformers Work: A Step-by-Step Worked Example

A tiny GPT-style model, small enough to follow by hand, from “I love AI” to the word it picks next

Every large language model you have used, GPT, Claude, Gemini, LLaMA, runs on one idea: the Transformer. Learn how Transformers work and you have learned the shared foundation of all of them. Production language models may contain hundreds of millions to hundreds of billions of parameters and usually have dozens to around one hundred layers. Their vocabularies may contain tens of thousands or even hundreds of thousands of tokens. You cannot realistically follow every calculation with a pencil.

So we shrink it until you can.

Nothing ahead needs more than arithmetic. It does assume you are comfortable with the idea of a model learning patterns from data, so if that part still feels shaky, our guide to machine learning for beginners covers the groundwork first. This article will still be here when you get back.

The clearest way to learn this machinery is to watch it run. So we build a Transformer small enough to follow number by number, then feed it three words:

I love AI

We follow every number, one operation at a time, until the model picks the next word. At each step you get three things: a plain explanation of what is happening and why, a look at how the real models (the original 2017 Transformer, GPT-2, GPT-3, LLaMA) do the same step at scale, and now and then a tip or a question to make it stick.

The numbers are tiny and made up. The machinery is the real thing.

First, a sense of scale

Here is the entire model we will build, next to models that write code and answer questions. They share the same core attention mechanism, but differ in architecture, normalization, positional encoding, and feed-forward design. The GPT-2 figures come from the original GPT-2 paper.

Our settings in one place:

Each token is described by six numbers. Attention is split into three heads, so each head works in

$$d_{\text{head}}=\frac{d_{\text{model}}}{n_{\text{heads}}}=\frac{6}{3}=2$$

dimensions. GPT-2 small makes the same split: 768 dimensions cut into 12 heads gives 64 per head. The arithmetic per head is identical to ours. There is just more of it.

Three family trees, one block

One column in that first table needs explaining before we start building. Why does the 2017 row say 6+6 layers, while every other row says a single number? Because Transformers come in three arrangements, and they are not interchangeable.

So 6+6 means six encoder layers followed by six decoder layers. The original paper was a translation model, so it needed both halves. Today’s chat models keep only the second half, which is why every other row in the table carries a single layer count.

We build the decoder-only kind, because that is what powers the models you actually talk to. Everything from Act 2 onward, especially the causal mask, follows from that choice.

Act 1: Turning words into numbers

A model cannot read letters. It reads numbers. Act 1 is entirely about that conversion.

Step 1 and 2: The vocabulary and token IDs

Think of the vocabulary as the model’s dictionary. We keep it small but relatable, seven everyday words that can form sensible little phrases, so the final prediction actually reads like something a person might say:

$$V=\{\text{I},\text{love},\text{AI},\text{and},\text{data},\text{math},\text{too}\}$$

Our tokenizer maps each word straight to an ID:

$$\text{I}\to 0,\text{love}\to 1,\text{AI}\to 2\Rightarrow [0,1,2]$$

From now on the model only sees [0, 1, 2].

In real models: a word may become one token, several subword tokens, or occasionally part of a token that includes surrounding spaces or punctuation. Real tokenizers (GPT uses one called BPE) split on frequent sub-word chunks, so “unbelievable” might become un, believ, able.  Because  GPT-2 uses byte-level BPE, it can represent virtually any Unicode string, even when a word was never present as a complete vocabulary token. Our clean one-word-per-ID mapping is the toy version of the same idea.

Step 3: Look up the embeddings

An ID like 2 is just a name tag. It carries no meaning. So we replace each ID with a small vector of numbers called an embedding. Picture it as a set of coordinates that place the word somewhere in “meaning space,” the way latitude and longitude place a city on a map.

We keep one vector for every word in an embedding table $E$ of shape $7\times 6$ (seven words, six numbers each). In a real model these numbers are learned during training. Here we pick simple ones so we can do the math by hand:

Looking up [0, 1, 2] just means grabbing rows 0, 1, and 2:

$$X_{\text{emb}}=\left[\begin{array}{cccccc}1& 0& 1& 0& 0& 0\\ 0& 1& 1& 0& 0& 0\\ 1& 1& 0& 1& 0& 0\end{array}\right](3\times 6)$$

Rows are tokens, columns are features.

In real models: this table is enormous. GPT-2’s embedding matrix is $\mathrm{50,257}\times 768$, which is about 38 million numbers just to represent words, before any “thinking” happens. Same lookup, far more rows and columns.

Step 4: Add positional encoding

Here is a problem. Attention, which we build in Act 2, compares words to each other but has no built-in sense of order. To it, “Dog bites man” and “Man bites dog” look like the same bag of words. Same words, opposite meaning.

The fix is to stamp each position with its own signature and add it to the embedding, like writing a seat number on every ticket. For readable arithmetic we use flat values:

$${\text{pos}}_0=[0,0,0,0,0,0],{\text{pos}}_1=[0.1,\dots ,0.1],{\text{pos}}_2=[0.2,\dots ,0.2]$$

In real models: the choice of “seat number” is a whole research topic.

• The original 2017 paper used fixed sine and cosine waves, with one frequency shared by each pair of dimensions (sine on the even index, cosine on the odd): $$P{E}_{(pos,2i)}=\sin (\frac{pos}{{10000}^{2i/d_{\text{model}}}}),P{E}_{(pos,2i+1)}=\cos (\frac{pos}{{10000}^{2i/d_{\text{model}}}})$$
• GPT-2 dropped the formula and simply learned a position table (a $1024\times 768$ matrix), one row per slot. Those dimensions are for GPT-2 small; the larger variants keep the same 1024 slots but a wider embedding.
• Many modern decoder-only models, including LLaMA, use Rotary Position Embeddings (RoPE). RoPE rotates components of the Query and Key vectors so their dot products encode relative-position information. Long-context performance still depends on the training context length and the specific RoPE scaling method used.

We use plain $0.1$ steps to keep the sums clean, but they play the exact same role: telling the model where each word sits.

Step 5: Combine into the input matrix X

Now add each position vector to its word’s embedding. Meaning plus place, in one vector:

• I (pos 0): $[1,0,1,0,0,0]+[0,0,0,0,0,0]=[1,0,1,0,0,0]$
• love (pos 1): $[0,1,1,0,0,0]+[0.1,0.1,0.1,0.1,0.1,0.1]=[0.1,1.1,1.1,0.1,0.1,0.1]$
• AI (pos 2): $[1,1,0,1,0,0]+[0.2,0.2,0.2,0.2,0.2,0.2]=[1.2,1.2,0.2,1.2,0.2,0.2]$

$$X=\left[\begin{array}{cccccc}1& 0& 1& 0& 0& 0\\ 0.1& 1.1& 1.1& 0.1& 0.1& 0.1\\ 1.2& 1.2& 0.2& 1.2& 0.2& 0.2\end{array}\right](3\times 6)$$

This matrix, words fused with positions, is what enters attention.

Act 2: Self-attention, where words start talking

This is the heart of the Transformer, and the part of how Transformers work that most explanations rush, so we go slowly. What follows is self-attention explained one operation at a time. The one-line intuition: every word looks at the other words it is allowed to see, decides how relevant each one is, and pulls in a blend of their information.

The idea behind Query, Key, and Value

Attention borrows the logic of a search engine. Each word produces three vectors:

• Query (Q): what am I looking for? (your search box text)
• Key (K): what do I advertise about myself? (the label on each result)
• Value (V): what do I actually hand over if you pick me? (the content behind the link)

A word’s Query is compared against every word’s Key to score relevance. Those scores become weights, and the weights pull together everyone’s Values. That is attention in one breath.

Each head has its own learned matrices $W_Q,W_K,W_V$. Because $X$ is $3\times 6$ and each head uses 2 dimensions, each matrix is $6\times 2$, so

$$(3\times 6)(6\times 2)=3\times 2\Rightarrow Q,K,V\text{ are each }3\times 2.$$

We run Head 1 in full detail. Then Heads 2 and 3 get the short treatment, because they are the same recipe with different weights, and repeating every multiply would only bore you.

Head 1, step A: the weights and the Q, K, V vectors

Our toy weights simply select two of the six input features for each of Q, K, and V. Head 1 reads features 0 and 1 for its Query, 0 and 2 for its Key, 1 and 3 for its Value:

$$W_Q^{(1)}=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 0& 0\\ 0& 0\\ 0& 0\\ 0& 0\end{array}\right],W_K^{(1)}=\left[\begin{array}{cc}1& 0\\ 0& 0\\ 0& 1\\ 0& 0\\ 0& 0\\ 0& 0\end{array}\right],W_V^{(1)}=\left[\begin{array}{cc}0& 0\\ 1& 0\\ 0& 0\\ 0& 1\\ 0& 0\\ 0& 0\end{array}\right]$$

Multiply $X$ by each one. For example, the Query for I is $[1,0,1,0,0,0]W_Q^{(1)}=[1,0]$. All three tokens:

$$Q_1=\left[\begin{array}{cc}1& 0\\ 0.1& 1.1\\ 1.2& 1.2\end{array}\right],K_1=\left[\begin{array}{cc}1& 1\\ 0.1& 1.1\\ 1.2& 0.2\end{array}\right],V_1=\left[\begin{array}{cc}0& 0\\ 1.1& 0.1\\ 1.2& 1.2\end{array}\right]$$

In real models: these are the workhorse weights, learned from data (not hand-picked selectors). GPT-3’s are far larger, but each still just turns a token vector into a Query, a Key, and a Value. Nothing more exotic than the multiply you see here.

Head 1, step B: attention scores Q1 K1 transpose

To score how much one word cares about another, take the dot product of the first word’s Query with the second word’s Key. A big dot product means “these two match.” We transpose $K_1$ so the shapes line up: $(3\times 2)(2\times 3)=3\times 3$.

Row = the word asking. Column = the word being checked. For instance, love checking love is $[0.1,1.1]\cdot [0.1,1.1]=0.01+1.21=1.22$:

$$Q_1{K}_1^{\mathrm{\top }}=\left[\begin{array}{ccc}1& 0.1& 1.2\\ 1.2& 1.22& 0.34\\ 2.4& 1.44& 1.68\end{array}\right]$$

Head 1, step C: scale by square root of d_k

There is a subtle trap. As the head dimension grows, dot products grow with it. Large scores can make softmax excessively sharp and produce very small gradients, making optimization harder. The paper’s fix is simple: divide by $\sqrt{d_k}$. Think of it as turning the volume down to a sane level. Here $d_k=2$, so we divide by $\sqrt{2}\approx 1.414$:

$$S_1=\frac{Q_1{K}_1^{\mathrm{\top }}}{\sqrt{2}}=\left[\begin{array}{ccc}0.707& 0.071& 0.849\\ 0.849& 0.863& 0.240\\ 1.697& 1.018& 1.188\end{array}\right]$$

In real models: identical formula. In the original Transformer $d_k=64$, so it divides by $\sqrt{64}=8$. The number changes, the reason does not.

Head 1, step D: mask the future

We are building a decoder, the GPT family. Its job is to predict the next word, so during training it must never peek ahead. A word may look only at itself and the words before it:

• I can see: I
• love can see: I, love
• AI can see: I, love, AI

Picture sliding a blank card down a page while reading, covering everything ahead. We do this with math by adding a mask that pushes forbidden spots to negative infinity, so they vanish after softmax:

$$M=\left[\begin{array}{ccc}0& -\mathrm{\infty }& -\mathrm{\infty }\\ 0& 0& -\mathrm{\infty }\\ 0& 0& 0\end{array}\right]\Rightarrow S_1+M=\left[\begin{array}{ccc}0.707& -\mathrm{\infty }& -\mathrm{\infty }\\ 0.849& 0.863& -\mathrm{\infty }\\ 1.697& 1.018& 1.188\end{array}\right]$$

Head 1, step E: softmax into attention weights

Now turn raw scores into a clean split of attention. Imagine each word has exactly 100% attention to spend. Softmax decides how to divide it, and every row adds to 1:

$$\text{softmax}(x_i)=\frac{e^{x_i}}{\sum _j{e}^{x_j}}$$

Row by row (a lone visible score gets weight 1, and $-\mathrm{\infty }$ gets 0):

$$A_1=\left[\begin{array}{ccc}1& 0& 0\\ 0.496& 0.504& 0\\ 0.474& 0.241& 0.285\end{array}\right]$$

Read the last row: when predicting after AI, this head splits its attention across all three visible words, leaning most on I.

Head 1, step F: mix the Values

Finally, use those weights to blend the Values, $H_1=A_1{V}_1$. Each output row is a weighted mixture. For AI:

$$0.474[0,0]+0.241[1.1,0.1]+0.285[1.2,1.2]=[0.607,0.366]$$

$$H_1=\left[\begin{array}{cc}0& 0\\ 0.554& 0.050\\ 0.607& 0.366\end{array}\right](3\times 2)$$

That is one complete attention head. Every word now carries a little information borrowed from the words it was allowed to see.

Head 2: same recipe, different features

Head 2 repeats every step above, but reads features 2 and 3 for its Query, 1 and 3 for its Key, 0 and 2 for its Value:

$$W_Q^{(2)}=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 1& 0\\ 0& 1\\ 0& 0\\ 0& 0\end{array}\right],W_K^{(2)}=\left[\begin{array}{cc}0& 0\\ 1& 0\\ 0& 0\\ 0& 1\\ 0& 0\\ 0& 0\end{array}\right],W_V^{(2)}=\left[\begin{array}{cc}1& 0\\ 0& 0\\ 0& 1\\ 0& 0\\ 0& 0\\ 0& 0\end{array}\right]$$

Running the same chain (build $Q_2,K_2,V_2$, score, scale, mask, softmax) gives:

$$A_2=\left[\begin{array}{ccc}1& 0& 0\\ 0.297& 0.703& 0\\ 0.180& 0.229& 0.591\end{array}\right],H_2=A_2{V}_2=\left[\begin{array}{cc}1& 1\\ 0.367& 1.070\\ 0.912& 0.550\end{array}\right]$$

Notice the contrast with Head 1. In Head 1, love split its attention almost evenly between “I” and “love.” In Head 2, love leaned mostly on itself (0.703). Same sentence, different focus.

Head 3: the sleepy head

Head 3 reads the newer position-heavy features, 4 and 5 for its Query, 3 and 4 for its Key, 2 and 5 for its Value:

$$W_Q^{(3)}=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 0& 0\\ 0& 0\\ 1& 0\\ 0& 1\end{array}\right],W_K^{(3)}=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 0& 0\\ 1& 0\\ 0& 1\\ 0& 0\end{array}\right],W_V^{(3)}=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 1& 0\\ 0& 0\\ 0& 0\\ 0& 1\end{array}\right]$$

$$Q_3=\left[\begin{array}{cc}0& 0\\ 0.1& 0.1\\ 0.2& 0.2\end{array}\right],K_3=\left[\begin{array}{cc}0& 0\\ 0.1& 0.1\\ 1.2& 0.2\end{array}\right],V_3=\left[\begin{array}{cc}1& 0\\ 1.1& 0.1\\ 0.2& 0.2\end{array}\right]$$

Because this head reads the small positional features, its raw scores are tiny (the largest is only $0.28$), so after softmax it spreads attention almost evenly:

$$A_3=\left[\begin{array}{ccc}1& 0& 0\\ 0.496& 0.504& 0\\ 0.308& 0.317& 0.375\end{array}\right],H_3=A_3{V}_3=\left[\begin{array}{cc}1& 0\\ 1.050& 0.050\\ 0.731& 0.107\end{array}\right]$$

Act 3: Combining the heads

Step: Concatenate

Each word now holds two numbers from each of the three heads. Glue them side by side, in order, back to a $3\times 6$ matrix, one six-number vector per word:

$$H=\left[\begin{array}{cccccc}0& 0& 1& 1& 1& 0\\ 0.554& 0.050& 0.367& 1.070& 1.050& 0.050\\ 0.607& 0.366& 0.912& 0.550& 0.731& 0.107\end{array}\right](3\times 6)$$

Step: Output projection W_o

Gluing is not enough. The three heads worked in separate lanes and never compared notes. A learned matrix $W_O$ (shape $6\times 6$) lets them mix, so a discovery in one head can influence another’s contribution. To keep the arithmetic clean, we use the identity matrix, which passes the numbers through untouched:

$$O=H{W}_O=H$$

In real models: $W_O$ is fully learned and does real mixing. Setting it to identity is the one place our toy visibly cheats, so keep in mind that in a real model this step matters.

Act 4: Add and Norm, then Feed-Forward

Step: First residual connection (the “Add”)

Here is a small trick with huge consequences. A residual connection just adds the block’s input back onto its output:

$$R_1=X+O$$

• I: $[1,0,1,0,0,0]+[0,0,1,1,1,0]=[1,0,2,1,1,0]$
• love: $[0.1,1.1,1.1,0.1,0.1,0.1]+[0.554,0.050,0.367,1.070,1.050,0.050]=[0.654,1.150,1.467,1.170,1.150,0.150]$
• AI: $[1.2,1.2,0.2,1.2,0.2,0.2]+[0.607,0.366,0.912,0.550,0.731,0.107]=[1.807,1.566,1.112,1.750,0.931,0.307]$

Why add the input back? The residual connection gives both the original representation and the gradient a direct path through the block. Attention adds new context on top of the word rather than replacing it.

Step: First layer normalization (the “Norm”)

After all that adding, the numbers can drift to different scales. LayerNorm rescales each word’s vector on its own to a steady mean of 0 and variance of 1 (with $\gamma =1$, $\beta =0$, ignoring the tiny $ϵ$). It is the z-score you may know from statistics, applied per word:

$$\text{LayerNorm}(x)=\frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}}$$

Take token I $=[1,0,2,1,1,0]$. Its mean is $\mu =5/6\approx 0.833$, so centered it is $[0.167,-0.833,1.167,0.167,0.167,-0.833]$. Its variance is $0.472$, so $\sigma \approx 0.687$, and dividing gives $[0.243,-1.213,1.698,0.243,0.243,-1.213]$. All three tokens:

$$Y=\left[\begin{array}{cccccc}0.243& -1.213& 1.698& 0.243& 0.243& -1.213\\ -0.701& 0.447& 1.179& 0.493& 0.447& -1.865\\ 1.064& 0.608& -0.253& 0.957& -0.595& -1.779\end{array}\right]$$

In real models: the placement of this step evolved. The 2017 paper normalized after the residual add (post-norm). GPT-2 moved it to before each sub-layer (pre-norm) because it trains more stably when deep. LLaMA-style models swap LayerNorm for a lighter cousin called RMSNorm. The goal never changes: keep the numbers well-behaved.

Step: Feed-forward network

Attention let words share information. The feed-forward network (FFN) now lets each word think on its own, with no looking at neighbors. Picture each word stepping into a private booth to process what it just heard:

$$\text{FFN}(x)=\text{ReLU}(x{W}_1+b_1)W_2+b_2$$

We use identity weights and zero biases, so the only real action is ReLU, which zeroes out negatives and keeps positives:

$$\text{ReLU}(x)=max(0,x)$$

• I: $[0.243,-1.213,1.698,0.243,0.243,-1.213]\to [0.243,0,1.698,0.243,0.243,0]$
• love: $[-0.701,0.447,1.179,0.493,0.447,-1.865]\to [0,0.447,1.179,0.493,0.447,0]$
• AI: $[1.064,0.608,-0.253,0.957,-0.595,-1.779]\to [1.064,0.608,0,0.957,0,0]$

$$F=\left[\begin{array}{cccccc}0.243& 0& 1.698& 0.243& 0.243& 0\\ 0& 0.447& 1.179& 0.493& 0.447& 0\\ 1.064& 0.608& 0& 0.957& 0& 0\end{array}\right]$$

In real models: the FFN is where a surprising share of the parameters live, because it first expands the dimension, then shrinks it back. The 2017 paper went $512\to 2048\to 512$ (four times wider in the middle). GPT-2 does $768\to 3072\to 768$. And the activation has evolved: the original used ReLU, GPT-2 uses GELU (a smoother ReLU), and LLaMA uses SwiGLU. We kept a flat $6\to 6$ with ReLU so you can see the shape of the idea without the extra numbers.

Step: Second residual and second LayerNorm

Add the FFN’s input back ($R_2=Y+F$), the same “keep the original” trick as before:

• I: $[0.485,-1.213,3.395,0.485,0.485,-1.213]$
• love: $[-0.701,0.894,2.358,0.986,0.894,-1.865]$
• AI: $[2.127,1.215,-0.253,1.913,-0.595,-1.779]$

Then normalize each word once more. The output of the whole Transformer block is:

$$Y_{\text{final}}=\left[\begin{array}{cccccc}0.053& -1.051& 1.945& 0.053& 0.053& -1.051\\ -0.833& 0.344& 1.426& 0.412& 0.344& -1.693\\ 1.190& 0.547& -0.487& 1.039& -0.728& -1.562\end{array}\right]$$

Each row is now a context-aware version of its word. Row 1 is “I” after hearing nothing before it, row 2 is “love” after hearing “I,” and row 3 is “AI” after hearing the whole phrase. That is one full decoder block done.

Act 5: Predicting the next word

Everything so far has been preparation. This is where we finally see how GPT predicts the next word, by turning one vector into a score for every word the model knows, and those scores into probabilities.

Step: Take the last word’s vector

To predict what comes after AI, we only need the last row. Thanks to the causal mask, it has already soaked up everything before it:

$$h_{\text{AI}}=[1.190,0.547,-0.487,1.039,-0.728,-1.562](1\times 6)$$

Step: Project into vocabulary scores

Here is the key idea that is easy to miss: at this point, we do not pick a word yet. The model gives a raw score to every word in the vocabulary, all seven of them, and then converts those scores into probabilities. With greedy decoding, the token with the highest score is selected. With sampling, a lower-probability token may also be chosen.

Each score comes from one column of a projection matrix $W_{\text{vocab}}$ (shape $6\times 7$, one column per word). To score a word, line its column up against $h_{\text{AI}}$, multiply the two position by position, and add up the results. That multiply-and-add is a dot product. With zero bias, $z=h_{\text{AI}}{W}_{\text{vocab}}$, which gives seven raw scores called logits:

$$W_{\text{vocab}}=\left[\begin{array}{ccccccc}1& 0& 0& 1& 0& 0& 1\\ 0& 1& 0& 1& 0& 0& 0\\ 0& 0& 1& 0& 0& 1& 0\\ 0& 0& 0& 0& 1& 0& 1\\ 0& 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0& 1& 0\end{array}\right]$$

In our toy every column is only 0s and 1s, which makes each dot product easy: a $1$ keeps that feature (any number times 1 is itself) and a $0$ drops it (any number times 0 is 0). So a column just selects which features of $h_{\text{AI}}$ to add up. Keep this row handy, with each feature’s position labelled:

Now the total calculation, one line per word (the listed features are the rows where that word’s column has a $1$):

• I selects feature 0: $1.190=1.190$
• love selects feature 1: $0.547=0.547$
• AI selects feature 2: $-0.487=-0.487$
• and selects features 0 and 1: $1.190+0.547=1.737$
• data selects features 3 and 4: $1.039+(-0.728)=0.311$
• math selects features 2 and 5: $-0.487+(-1.562)=-2.049$
• too selects features 0 and 3: $1.190+1.039=2.229$

Stacking those seven results, in vocabulary order (I, love, AI, and, data, math, too), gives the logits:

$$z=[1.190,0.547,-0.487,1.737,0.311,-2.049,2.229]$$

We scored all seven words before knowing which would win. The biggest logit is $2.229$, which belongs to too, so that is the word the model is leaning toward. Logits are still raw scores, though (they can be negative and do not add to 1), so the next step turns them into clean percentages.

Where do the 0s and 1s in $W_{\text{vocab}}$ come from? In this toy, we chose them by hand to keep the arithmetic simple, and, honestly, so the result lands on a sensible word. A real model gets no such hand-holding: it learns this matrix during training, nudging the numbers over millions of examples until useful words naturally earn the high scores. Real models often reuse the embedding table from Step 3 as this projection (a trick called weight tying), which saves millions of parameters and keeps reading and writing words consistent.

Step: Final softmax into probabilities

One last softmax turns the seven logits into probabilities that add to 1:

$$P_i=\frac{e^{z_i}}{\sum _j{e}^{z_j}}$$

The winner is too, and the runner-up and also reads naturally. Our tiny Transformer predicts:

I love AI → too (“I love AI too”)

This time the completion actually makes sense, but do not read too much into that. We chose every weight by hand, so the model has learned nothing; we simply picked a relatable vocabulary and a projection that lands on a sensible word. In a real model, training with gradient descent nudges every embedding and weight until sensible words rise to the top on their own. The machine you just watched uses the same core attention, residual, normalization, and feed-forward ideas found inside GPT. Real GPT models arrange some of these operations differently, most notably by using pre-normalization.

The whole flow on one page

Here is the whole transformer step by step, compressed into one diagram. Every arrow below is something we calculated by hand.

"I love AI"
    |  tokenize
[0, 1, 2]
    |  embedding lookup            (3 x 6)
    |  + positional encoding       X: 3 x 6
    v
Head 1: Q1 K1 V1   Head 2: Q2 K2 V2   Head 3: Q3 K3 V3
    |  QK^T  ->  scale by sqrt(dk)  ->  mask future  ->  softmax  ->  x V
Head 1 (3x2)   Head 2 (3x2)   Head 3 (3x2)
    |  concatenate                 3 x 6
    |  x W_O  (mix the heads)
    |  + residual  ->  LayerNorm
    |  feed-forward (ReLU)
    |  + residual  ->  LayerNorm    Y_final: 3 x 6
    |  take last-word row           h_AI: 1 x 6
    |  x W_vocab                    7 logits
    |  softmax                      7 probabilities
Predicted next word: "too"

Shape cheat-sheet

One-line summary of each part

• Embedding: turn a word ID into a vector of features.
• Positional encoding: stamp where each word sits, since attention ignores order.
• Query, Key, Value: what a word seeks, what it advertises, what it hands over.
• $Q{K}^{\mathrm{\top }}$: measure how strongly every Query matches every Key.
• Scaling: keep scores from blowing up before softmax.
• Causal mask: stop a word from reading the future.
• Softmax: split a fixed budget of attention into weights that sum to 1.
• Weights times Value: build a weighted blend of information.
• Multiple heads: learn several kinds of relationships in parallel.
• Concatenate plus $W_O$: join the heads, then let them mix.
• Residual connection: keep the original signal, add the new.
• LayerNorm: keep the numbers stable and comparable.
• Feed-forward: let each word process on its own.
• Vocabulary projection plus softmax: score every possible next word, then turn scores into probabilities.

The one equation to remember

Strip away the scaffolding and the entire attention mechanism is this:

$$\text{Attention}(Q,K,V)=\text{softmax}(\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}+M)V$$

Read it left to right: compare Query with Key, scale the scores, block the future with mask $M$, soften into weights, combine the Values. Everything else in this article is plumbing around that single line.

The takeaway

A Transformer does not store human meanings like “love is an emotion.” It stores numbers. Through repeated multiplication, attention, and a few nonlinear steps, refined by training, those numbers come to encode patterns that behave like understanding.

The whole forward pass fits in one sentence:

Turn words into vectors, let every allowed word examine the others, gather the useful information, transform it, and turn the final vector into probabilities for the next word.

You have now seen how Transformers work at the level of individual numbers. We walked through all of it, one calculation at a time, for I love AI, and out came too, creating the phrase I love AI too. Scale this system to billions of parameters and train it on vast amounts of text, and you get models that write code and hold conversations. The same core Transformer math remains, combined with architectural, optimization, and training improvements.

Transformers are one idea among several that make modern AI work. If you want the map rather than the machinery, our guide to AI fundamentals places this architecture next to the other pieces.

Want to go deeper?

If you want to push further into how Transformers work, three places are worth your time, in the order I would take them.

The paper itself. Attention Is All You Need (Vaswani and colleagues, 2017) is shorter and more readable than most people expect. Having just worked through the arithmetic by hand, you will recognise almost every equation in it.

The visual version. Jay Alammar’s The Illustrated Transformer explains the same architecture with pictures instead of numbers. If any step here felt abstract, seeing it drawn often finishes the job.

Build a real one. Andrej Karpathy’s nanoGPT is a working GPT in a few hundred lines of readable code. Everything you followed here, from embeddings and heads to masks, residuals and LayerNorm, is in there doing it for real.