Transformers and self-attention: the foundation of every LLM
TL;DR
In a Transformer each token decides what to attend to (Q·K) and pulls information back (V). Understanding this explains why long context is expensive, why prompt caching works and why position in the prompt matters.
Every LLM you use daily, from Claude to GPT and Gemini, is built on the Transformer architecture introduced in 2017 in the paper "Attention Is All You Need" (Vaswani et al., Google). Before that, NLP relied on recurrent networks (RNN/LSTM) that read one word at a time in order, which was slow and tended to "forget" distant information. The Transformer drops recurrence entirely in favour of self-attention: every token looks directly at every other token at once. Understanding this mechanism explains why long context is expensive and why where information sits in a prompt matters.
Self-attention in one sentence
For each token, the model asks: "To understand myself, which other tokens should I pay attention to, and how much?" In "The cat didn't eat because it was full", attention lets the token "it" put high weight on "the cat".
Query, Key, Value
Each token is projected into three vectors:
| Vector | Role | Analogy |
|---|---|---|
| Query (Q) | What this token is looking for | What you type into a search box |
| Key (K) | What this token "advertises" it contains | The title of each document |
| Value (V) | The content actually retrieved | The document body |
The attention score is the dot product Q·K, divided by the square root of the dimension for stability, turned into weights by softmax, then multiplied by V:
Attention(Q, K, V) = softmax(Q·Kᵀ / √d_k) · VThe remaining pieces
- Multi-head attention: several attention "heads" run in parallel, each learning a kind of relationship (syntax, coreference, position…), then their outputs are combined.
- Positional encoding: attention has no built-in sense of order, so position information is added to each token. Modern models use variants such as RoPE.
- Feed-forward + residual + layer norm: after attention, each token passes through a small network; skip connections and normalisation make it possible to stack dozens to hundreds of layers.
- Causal mask (decoder-only): text-generating LLMs only let tokens look backwards, never "peek" at future tokens. That's why models generate one token at a time.
Why this matters to users
| Architectural property | Practical consequence |
|---|---|
| Attention compares every pair of tokens: cost grows quadratically with length | Long context is slow and expensive; providers bill per input token |
| Tokens are generated one by one, each depending on what came before | Long outputs increase latency linearly; asking for brevity is the cheapest speed-up |
| Keys/Values of the prompt prefix can be reused (KV cache) | This is the basis of prompt caching: keep the start of the prompt stable to get the discount |
| Attention weight is spread unevenly over long contexts | Information gets "lost" in the middle; structure and position in the prompt matter |
Risks of misunderstanding
- "Attention = understanding": attention weights are not a reliable explanation of a model's decision; don't use attention heatmaps as audit evidence.
- "Longer context is always better": a big window doesn't mean the model uses every part equally. Selection still beats stuffing.
- Training from scratch: understanding the architecture doesn't mean you should train your own. For most projects, good prompts + RAG + an API are enough.
How to apply it
- Dev: Read Jay Alammar's "The Illustrated Transformer" (step-by-step diagrams), then implement attention in 30 lines of NumPy to really get it.
- Tech Lead: Use your understanding of the KV cache to design prompts: put fixed parts (system, documents, tools) first and keep them unchanged to benefit from caching.
- PM: When the team proposes "doubling the context", ask how much cost and latency go up, and whether better data selection is possible.
Sources & references
- Attention Is All You Need — Vaswani et al. (arXiv)
- The Illustrated Transformer — Jay Alammar