Transformers have quietly become the load-bearing infrastructure of modern AI. GPT-4, Claude, Gemini, Stable Diffusion, AlphaFold 2, Whisper — all of them are built on variants of a single architecture introduced in a 2017 Google paper titled "Attention Is All You Need." If you want to use AI tools with genuine competence rather than just operational intuition, understanding how transformers work is non-negotiable. It's the difference between knowing how to drive and understanding enough about engines that you can diagnose problems, anticipate limits, and make informed decisions about which vehicle to buy.
This guide builds that understanding from the ground up. We'll cover what problem transformers actually solved, how the core mechanisms work, what the major variants do differently, and where the architecture is heading. You don't need a math degree. You need patience and a willingness to think carefully about how information flows through a system. By the end, you'll be able to read model cards, evaluate capability claims, and make better decisions about which models to deploy — and why.
The Problem Transformers Were Built to Solve
Before 2017, sequence modeling — understanding and generating language, audio, or any ordered data — depended almost entirely on recurrent neural networks (RNNs) and their improved descendants, LSTMs and GRUs. These architectures processed sequences one token at a time, passing a hidden state forward like a baton. That worked, but it created two serious problems.
Sequential bottleneck. Because each step depended on the previous one, training couldn't be parallelized across a sequence. Processing a sentence of 100 words meant 100 sequential steps. On modern hardware built for parallel computation, this was like using a highway one lane at a time.
Vanishing memory. By the time an RNN reached token 80 in a sequence, the signal from token 1 had been compressed, diluted, and partially overwritten dozens of times. Long-range dependencies — the connection between a pronoun and its antecedent five paragraphs earlier, or a theme introduced in chapter one and resolved in chapter ten — were difficult to preserve.
Transformers eliminated both problems by replacing sequential processing with a mechanism called self-attention, which computes relationships between all tokens simultaneously. Every word can attend to every other word in a single pass. This is why transformers scaled so dramatically when researchers simply added more data and compute — the architecture was parallelizable from the start.
For a broader grounding in why this matters for the models you use daily, How Generative AI Works: The Questions Everyone Asks, Answered is a useful companion read.
Tokens: The Unit of Meaning
Before diving into the architecture itself, you need to understand what transformers actually process. Raw text doesn't go in. Tokens do.
A tokenizer breaks text into subword units — fragments that balance vocabulary size against representational flexibility. The word "transformers" might be one token. "Unhappiness" might be two: "un" and "happiness." A typical model like GPT-4 uses roughly 100,000 tokens in its vocabulary. As a rough rule: 1,000 words ≈ 750–850 tokens, though this varies significantly by language and content type.
Why this matters practically:
- Context windows are measured in tokens, not words. A 128,000-token context is roughly 90,000–100,000 words.
- Pricing for API calls is token-based. Knowing token ratios lets you estimate cost before running experiments.
- Model behavior is affected by tokenization quirks. Some models struggle with tasks requiring character-level reasoning (counting letters in a word, for instance) because they never "see" individual characters — only token chunks.
The Core Architecture: Encoder, Decoder, or Both
The original transformer in "Attention Is All You Need" had two major components: an encoder that reads and represents input, and a decoder that generates output. Most modern models use only one of these components, optimized for a specific category of task.
Encoder-Only Models
Encoder models (BERT and its descendants) read an entire input sequence at once, bidirectionally. Every token can attend to every other token — left and right — producing rich contextual representations. These are ideal for tasks where you need to understand text rather than generate it: classification, sentiment analysis, named entity recognition, semantic search.
The trade-off: encoder-only models can't generate text autoregressively. They're discriminative, not generative.
Decoder-Only Models
This is the dominant architecture for large language models today — GPT-4, Claude, LLaMA, Mistral. Decoder models generate text left to right, one token at a time. Each new token can only attend to tokens before it (masked self-attention), which makes generation coherent and prevents the model from "cheating" during training by looking ahead at the answer.
Decoder-only models have proven capable of handling understanding tasks too, especially at scale — which is why most frontier labs have converged on this design.
Encoder-Decoder Models
Models like T5, BART, and mT5 preserve the full original architecture. The encoder processes input; the decoder generates output conditioned on the encoder's representation. These architectures excel at tasks with a clear input-to-output mapping: translation, summarization, and structured data generation.
Self-Attention: The Mechanism That Changed Everything
Self-attention is the intellectual core of the transformer. Here's what it actually does.
Each token in a sequence is represented as a vector of numbers (an embedding). Self-attention transforms each token's embedding into three derived vectors: a Query (Q), a Key (K), and a Value (V). Think of it like a retrieval system:
- The Query is the question this token is asking: "What context do I need?"
- The Keys are the labels each other token broadcasts: "Here's what I'm about."
- The Values are the actual content each token contributes if selected.
The model computes a dot product between each token's Query and every other token's Key. Higher dot products mean higher relevance. These scores are scaled, passed through a softmax function to produce a probability distribution, and then used to create a weighted sum of the Values. The result is a new representation of each token — one that incorporates context from the most relevant other tokens in the sequence.
This all happens in parallel, across the entire sequence, in a single matrix operation. That's why transformers are so hardware-efficient on GPUs and TPUs.
Multi-Head Attention
One attention head captures one type of relationship. Multi-head attention runs several attention mechanisms in parallel, each with its own Q, K, V weight matrices. Different heads learn to attend to different phenomena: one might track syntactic subject-verb agreement; another might track coreference; another might track semantic similarity. Their outputs are concatenated and projected back into the main embedding dimension.
Most base transformer models use 8–96 attention heads depending on model size. Larger models use more heads with larger head dimensions.
Positional Encoding: Teaching the Model Where Things Are
Self-attention is order-agnostic by default. Shuffle the tokens in a sentence and the attention scores change, but the mechanism itself has no inherent sense of sequence. Transformers solve this by injecting positional information into each token's embedding before attention runs.
Absolute Positional Encodings
The original paper used fixed sinusoidal functions — a specific mathematical pattern that gives each position a unique fingerprint. Many early BERT-style models learned positional embeddings from scratch during training.
Relative and Rotary Positional Encodings
Absolute positions become problematic when you need to generalize beyond training sequence length. Newer techniques encode relative distance between tokens rather than absolute position. Rotary Position Embedding (RoPE), used by LLaMA and many modern models, encodes position by rotating the query and key vectors in a mathematically elegant way. This has shown better length generalization and has become the dominant approach in frontier models.
Understanding positional encoding helps explain why models have context limits and why extending those limits requires architectural changes, not just more training data.
Feed-Forward Networks and Layer Normalization
After each attention sub-layer, transformer blocks include a position-wise feed-forward network (FFN) — two linear layers with a nonlinear activation (typically ReLU or GELU) in between. This FFN applies the same transformation independently to each token's representation.
If self-attention is where tokens communicate with each other, the FFN is where each token "thinks" in isolation. Research suggests FFNs act as a kind of factual memory — patterns in these weight matrices encode much of the world knowledge that emerges during training.
Layer normalization is applied before or after each sub-layer (the "pre-norm" variant is now standard in most large models). It stabilizes training by keeping activation distributions in check, which becomes increasingly important as model depth increases into hundreds of layers.
Scale, Emergent Capability, and Why Size Matters
One of the most important empirical findings of the past six years is that transformer performance scales predictably with compute, data, and parameter count — a finding formalized in "scaling laws" research. Roughly speaking, doubling model size at constant compute budget yields consistent, measurable gains on most benchmarks.
More surprising: certain capabilities appear to emerge only above specific scale thresholds. Arithmetic, multi-step reasoning, in-context learning — these abilities are largely absent in small models and appear suddenly (from an external measurement perspective) as models cross size thresholds. This isn't magic; it's likely that small models develop the prerequisite sub-skills at different rates, and capability only becomes detectable once all sub-skills are present.
This has direct implications for practitioners: a model that failed at a task six months ago may succeed today not because of a different architecture, but simply because it's larger and trained on more data. The Complete Guide to Neural Networks covers the underlying learning mechanisms that make this scaling behavior possible.
Major Transformer Variants and What They're Optimized For
The base architecture has been forked aggressively. Here are the variants you'll encounter most often and what distinguishes them.
Sparse Mixture of Experts (MoE): Instead of activating all model parameters for every token, MoE models route each token through a learned selection of "expert" sub-networks. Mistral's Mixtral 8x7B and GPT-4 (reportedly) use this approach. Result: much larger total parameter counts with computation costs closer to a smaller dense model.
Flash Attention: Not a model variant but an algorithmic rewrite of the attention computation that's dramatically more memory-efficient by tiling the calculation to fit GPU memory hierarchy. This enabled context windows to expand from 4K to 128K+ tokens practically.
Grouped Query Attention (GQA): Reduces the number of unique Key and Value heads while keeping more Query heads, cutting memory bandwidth during inference significantly. Used in LLaMA 3 and most modern efficient models.
Vision Transformers (ViT): Applies the transformer architecture to images by treating image patches as tokens. Became competitive with convolutional networks at scale and is now the backbone of most multimodal models.
These variants matter when you're choosing between models for deployment. The How Generative AI Works Playbook covers how to evaluate models against real task requirements, which is where this architectural knowledge becomes operational.
What Transformers Still Can't Do Well
Intellectual honesty requires acknowledging the limits.
True recurrence and unbounded memory. Transformers operate on fixed context windows. They have no persistent memory between conversations unless you engineer it externally. Every token in the context window costs compute; there's no free lunch for longer contexts.
Strict logical and symbolic reasoning. Transformers learn statistical patterns, not formal rules. They approximate reasoning rather than perform it — which is why they can fail catastrophically on problems that require precise symbolic manipulation, even when they succeed on superficially similar problems.
Sample efficiency. Humans learn language from vastly less data. A child doesn't need 10 trillion tokens to acquire fluency. Transformers compensate for architectural inefficiency with brute-force data scale, which has environmental and economic costs.
Systematic generalization. Transformers tend to struggle when inputs require combining rules in genuinely novel ways not represented in training. They interpolate well; they extrapolate poorly.
These aren't reasons to avoid transformers — they remain the most capable general-purpose architecture available. But knowing the failure modes helps you design workflows that route around them rather than walk into them blind. Building a Repeatable Workflow for How Generative AI Works addresses exactly this kind of architectural awareness applied to practical process design.
Where the Architecture Is Heading
The transformer isn't finished evolving. Several directions are receiving serious research investment.
State Space Models (SSMs): Mamba and its successors process sequences through a recurrent-like mechanism that scales linearly with sequence length rather than quadratically. They've shown competitive performance with transformers on some benchmarks, particularly for very long sequences. Whether they'll displace transformers at frontier scale remains an open question.
Hybrid architectures: Rather than picking one approach, frontier labs are experimenting with models that interleave transformer attention layers with SSM layers, trying to capture the best of both.
Longer context and memory augmentation: Research into external memory stores, retrieval augmentation, and more efficient attention variants continues at pace. The goal is effective context that scales to book-length or beyond.
Multimodality as the default: The boundary between language models and vision models has already blurred significantly. Native multimodal training — where the model learns from interleaved text, image, audio, and video from the start — is likely to become standard practice. The Future of How Generative AI Works explores what this convergence looks like at the application layer.
Frequently Asked Questions
What does "transformer" mean in the context of AI?
In AI, a transformer is a neural network architecture that uses self-attention mechanisms to process sequences of data — most commonly text, but also images, audio, and video. The name comes from how the model "transforms" input representations into contextually enriched output representations, not from electrical transformers or fictional robots.
Why is the attention mechanism considered such a breakthrough?
Attention allows every element in a sequence to directly influence every other element in a single computational step, rather than passing information through a chain of sequential operations. This makes the architecture highly parallelizable on modern hardware and dramatically better at capturing long-range dependencies — two limitations that had plagued earlier architectures for years.
What is a context window and why does it matter?
A context window is the maximum number of tokens a model can process in a single forward pass — its working memory for a given interaction. Anything outside the context window is invisible to the model. Context length affects how much of a document the model can "see" at once, which directly impacts task quality on long documents, extended conversations, and complex multi-step workflows.
How is GPT different from BERT architecturally?
GPT uses a decoder-only transformer, which processes tokens left to right and is optimized for text generation. BERT uses an encoder-only transformer, which reads sequences bidirectionally and is optimized for understanding and classification tasks. Both use self-attention, but the masking strategy differs: GPT masks future tokens during training; BERT masks random tokens in both directions.
Do you need to understand transformers to use AI tools effectively?
You don't need to implement transformers to use AI tools. But understanding the architecture — context windows, tokenization, the difference between encoder and decoder models — makes you substantially better at prompt design, model selection, cost estimation, and diagnosing unexpected outputs. It's the difference between operating AI and understanding it.
What's the difference between parameters and tokens?
Parameters are the learned weights inside a model's neural network — the numbers adjusted during training that encode the model's knowledge and capabilities. Tokens are the units of input and output text the model processes at inference time. A 70-billion-parameter model has 70 billion internal numeric values; how many tokens it can process at once depends on its context window, which is a separate architectural specification.
Key Takeaways
- Transformers replaced sequential RNNs with parallelizable self-attention, enabling both faster training and better long-range dependency modeling.
- Tokens — not words — are the fundamental unit. Understanding token counts affects cost, context management, and prompt strategy.
- Encoder-only models (BERT) excel at understanding tasks; decoder-only models (GPT, Claude) excel at generation; encoder-decoder models (T5) excel at structured transformation tasks.
- Self-attention computes Query-Key dot products to determine relevance, then uses those weights to aggregate Values — all in parallel across the full sequence.
- Multi-head attention lets a single layer capture multiple types of linguistic and semantic relationships simultaneously.
- Transformer capabilities scale predictably with compute and data, and certain abilities emerge only above specific scale thresholds.
- Key architectural variants — MoE, Flash Attention, GQA, ViT — address specific trade-offs in efficiency, context length, and modality.
- Core limitations include fixed context windows, weak symbolic reasoning, and poor extrapolation beyond training distribution.
- The architecture is actively evolving toward hybrid designs, longer context, and native multimodality — understanding the base architecture positions you to evaluate these changes as they arrive.