From text to tokens to vectors, then from vectors through stacked attention layers — and finally to a model that predicts one token at a time.
A model never sees characters directly. Text is split by a tokenizer — usually byte-pair encoding (BPE) or a variant — into subword tokens. Common words become single tokens; rare words get broken into pieces. Each token has an integer ID in a fixed vocabulary (typically 50k–200k entries).
Token IDs are arbitrary integers, so the model needs a way to turn them into geometry. It learns an embedding matrix E of shape [vocab_size, d_model]. Row i is the vector for token i. So the token "cat" becomes a list of, say, 4096 floating-point numbers. Similar meanings end up near each other in this space because training pushes them there.
Positional information. Attention is permutation-invariant — it has no built-in sense of order. So a positional encoding is added to (or injected into) each token embedding. Modern models use rotary position embeddings (RoPE), which rotate query/key vectors by an angle proportional to position. This lets the model extrapolate to sequences longer than those seen in training, which is why context windows can grow from 4k to 128k+ tokens.
Embeddings are typically compared by cosine similarity — the angle between vectors, ignoring magnitude. Click a pair below to compare.
This is the core mechanism. Every token produces three vectors by multiplying its embedding x by learned weight matrices:
Attention scores how well each query matches every key, converts those scores into weights with softmax, then takes a weighted average of the values:
The /√d_k term keeps dot products from growing with dimension — otherwise softmax would saturate and gradients would vanish. The output for each token is a blend of information from all tokens it attended to. Multi-head attention runs several of these in parallel with different learned projections, so each head can specialize (one tracks syntax, another coreference, another adjacent words), and the results are concatenated and projected back.
Click any token to make it the query. The bars show how it distributes its attention across the sentence (illustrative weights).
A decoder-only LLM must not see the future. Before softmax, the model sets scores for positions after the current token to −∞, so those weights become zero. This is the causal mask, and it's what makes generation left-to-right while still training on the whole sequence in parallel.
Attention alone isn't enough. Each block wraps attention with a few stabilizing components, and a full model stacks dozens of these blocks.
This "Pre-LN" arrangement (normalize → sublayer → add residual) is standard today because it trains more stably than the original Post-LN formulation. Attention handles communication between tokens; the feed-forward network handles computation within each token.
How does the model learn those weights? Mostly by self-supervised next-token prediction — no human labels required, just raw text.
Given a sequence, the model predicts the token at every position from the tokens before it. The loss is the average negative log-probability of the correct next token (cross-entropy):
Because the causal mask hides the future, all positions are predicted in a single forward pass — massively parallel, which is why transformers train efficiently on GPUs. Do this on trillions of tokens and the model absorbs grammar, facts, reasoning patterns, and style.
Next-token prediction sounds trivial, but to do it well across the whole internet the model is forced to build internal representations of syntax, entities, arithmetic, and even multi-step reasoning. Capability emerges as a side effect of compressing prediction — that's the bet behind scaling.
Masked vs. causal. Some models (BERT-style) use masked language modeling — hide random tokens and predict them from both sides — great for understanding tasks and embeddings. Decoder-only causal models (GPT, Claude, Llama) predict left-to-right, which makes them natural generators. This explainer focuses on the causal variant.
Generation is a loop. The prompt is tokenized, embedded, and pushed through N blocks of attention + MLP. The final hidden state of the last token is projected onto the vocabulary (via a matrix often tied to the embedding matrix) to produce logits, which softmax turns into probabilities. One token is sampled and appended, and the process repeats — each pass conditioned on everything before it.
The one-sentence version. A transformer LLM turns tokens into vectors (embeddings + position), repeatedly lets every token gather relevant context from other tokens (attention) and compute on what it gathered (MLPs), and is trained by predicting the next token over enormous text — a simple objective that scales into general language ability.