From characters to coherent text — the architecture, the math, and the magic.
The pipeline end-to-end
Byte-level BPE
Tokens become vectors
Q, K, V & multi-head
Logits → next token
Text in → text out. Everything in between is matrix multiplication.
"The cat sat"
"on" (P = 0.42) "down" (P = 0.21) "in" (P = 0.11) …
All three families stack the transformer blocks we just saw. They differ in how attention sees the sequence — and that determines what they're good at. For the rest of this talk we'll focus on decoder-only, the architecture behind every modern chat / generation model.
BERT family
Sees: every token sees every other (full context).
Good at: classification, embeddings, search/retrieval, NER, fill-in-blank.
Bad at: generating text — no native left-to-right notion.
Examples: BERT, RoBERTa, DeBERTa, sentence-transformers
Original Transformer / T5 family
Sees: encoder bidirectional · decoder causal · decoder cross-attends to encoder.
Good at: translation, summarization, structured input → structured output.
Trade-off: ~2× the parameters & complexity for the same scale.
Examples: T5, FLAN-T5, BART, original Transformer (Vaswani 2017)
GPT family — modern LLMs
Sees: each token sees only past tokens (left-to-right).
Good at: text generation, chat, code, reasoning, agents — anything autoregressive.
Why it won: simpler, scales beautifully, one model handles all tasks via prompting.
Examples: GPT-4, Claude, Llama, Mistral, Gemini, Qwen, DeepSeek
The same model can do any NLP task by just prompting it differently — translation, classification, summarization, code, math, chat. No need to design separate architectures or fine-tune for each task. Combined with simpler training (one objective: predict next token), decoder-only proved easier to scale to billions then trillions of parameters. The other families still exist for specialized roles (embeddings, real-time translation), but every consumer-facing chatbot today is decoder-only.
Neural networks operate on numbers — not characters or words. Tokenization is the bridge.
"The cat sat on the mat."
[791, 8415, 7731, 389, 279, 5634, 13]
The model never sees "cat" — it sees the integer 8415.
The output will be another integer, which gets detokenized back to text.
The tokenization algorithm behind GPT, Llama, Claude, and most modern LLMs.
"unbelievable"
u n b e l i e v a b l e
[403, 8265, 480]
"The cat sat on the mat."
Note the leading spaces — whitespace is part of the token.
Not all tokens come from text. The vocabulary also reserves control tokens that structure the conversation, mark boundaries, and trigger behaviors.
messages = [
{"role": "system", "content": "You are helpful."},
{"role": "user", "content": "Hi!"},
{"role": "assistant", "content": "Hello!"}
]
<|begin_of_text|> <|start_header_id|>system<|end_header_id|> You are helpful.<|eot_id|> <|start_header_id|>user<|end_header_id|> Hi!<|eot_id|> <|start_header_id|>assistant<|end_header_id|> Hello!<|eot_id|>
[128000, 128006, 9125, 128007, 2675, 527, 11190, 13, 128009, 128006, 882, 128007, 13347, 0, 128009, 128006, 78191, 128007, 9906, 0, 128009]
IDs ≥ 128000 are special. The model learns: "after assistant<|end_header_id|>, generate a reply, then emit <|eot_id|> to stop."
System / user / assistant boundaries are just tokens. The model learns to respect them during fine-tuning.
Generation halts when the model emits <|eot_id|> or <|end_of_text|> — that's how the API knows the turn is done.
Function calling, vision inputs, and reasoning modes all get their own special tokens to switch the model's behavior.
Each token ID indexes into a learned table of high-dimensional vectors.
A matrix E ∈ ℝV × d where:
token_id → E[token_id] ∈ ℝ⁴⁰⁹⁶ 403 → [ 0.12, -0.84, 0.31, …, 0.07] 8265 → [-0.45, 0.22, 0.91, …,-0.18] 480 → [ 0.67, -0.13, 0.04, …, 0.55]
Plus a positional encoding is added so the model knows token order.
vec("king") − vec("man") + vec("woman") ≈ vec("queen")
vec("Paris") − vec("France") + vec("Italy") ≈ vec("Rome")
Direction in the space encodes meaning. Distance encodes similarity.
Self-attention is permutation-invariant — it sees a bag of tokens. Without position info, "the cat sat" and "sat cat the" would produce identical outputs.
Attention weights only depend on token content, not order. So we have to add the order back in explicitly.
x_i = embedding(token_i) + position_vector(i)
└─ what it is ─┘ └─── where it sits ───┘
Same shape as the embedding (ℝᵈ), so addition just works. The transformer now receives both signals fused into one vector.
Fixed sin/cos at exponentially varying frequencies. No parameters. Generalizes to unseen sequence lengths.
A second embedding table indexed by position (0…N−1). Trained via backprop. Cannot extrapolate past max length.
Rotary Position Embedding — rotates Q and K vectors by an angle proportional to position. Encodes relative distance directly, scales to long contexts.
Treat each position like a token. Allocate a second lookup table P ∈ ℝN_max × d and train it via backprop alongside everything else.
position_id → P[position_id] ∈ ℝᵈ 0 → [ 0.21, -0.74, 0.55, …, -0.08] ← learned 1 → [-0.13, 0.62, 0.41, …, 0.27] ← learned 2 → [ 0.08, -0.31, 0.92, …, -0.44] ← learned … N_max−1 → [ … ]
Each input vector becomes the sum of two learned lookups:
N_max × d parameters (e.g. 2048 × 4096 ≈ 8M).This is why modern LLMs (Llama, GPT-NeoX, Claude) moved to RoPE — same idea but encoded as a rotation, which generalizes to longer sequences.
Every token asks every other token: "how much should I pay attention to you?"
From each token's embedding x, three vectors are projected via learned weight matrices:
Q = x · W_Q ← "what am I looking for?" K = x · W_K ← "what do I offer?" V = x · W_V ← "what info do I carry?"
Attention(Q, K, V) = softmax( Q·Kᵀ / √d ) · V
Sentence: "The cat sat on the mat"
When processing "sat", it can only attend to itself and prior tokens.
new_representation = 0.10·V_The + 0.75·V_cat + 0.15·V_sat
(future V_on, V_the, V_mat are excluded)
Without this single scaling factor, attention would not train. Here's the math, the failure mode, and the fix.
Assume each component of Q and K is roughly independent with mean 0 and variance 1. Then for a single dot product:
q · k = Σᵢ qᵢ · kᵢ (i = 1 … d) E[q · k] = 0 Var[q · k] = d Std[q · k] = √d
So as d grows (typical: 64, 128, even 256 per head), the raw scores q·k grow proportionally to √d.
For d=128 → typical scores ≈ ±11. For d=4096 → typical scores ≈ ±64.
softmax( Q·Kᵀ / √d ) · V
Softmax is exponential. When inputs are large, one value dominates:
scores = [12, 8, 10, 11] softmax = [0.71, 0.01, 0.10, 0.18]
≈ one-hot. Gradient ≈ 0. Model can't learn.
scores = [1.06, 0.71, 0.88, 0.97] softmax = [0.30, 0.21, 0.25, 0.27]
Soft distribution. Gradients flow. Model learns.
Dot products grow with dimension. Softmax of large numbers is brittle. /√d keeps the input to softmax dimension-independent — so the same architecture works whether you have 64 or 4096 features per head.
LLMs are trained to predict the next token. If a token could see future tokens, training would be trivial — and inference would break.
After computing Q·Kᵀ / √d but before softmax, add a mask matrix M:
scores = Q·Kᵀ / √d scores = scores + M ← M is upper-triangular −∞ attn = softmax(scores) ← e^−∞ = 0, so future is gone output = attn · V
Setting masked entries to −∞ (not 0) means after softmax they become exactly 0 — and the visible probabilities still sum to 1.
Each row = one query token. Each column = a key it might attend to. Green = allowed, red = masked (−∞).
Encoder models (BERT) use bidirectional attention — every token sees every other token. Decoder LLMs (GPT, Llama, Claude) use causal attention — each token sees only the past. That's the only architectural difference that makes one a generator and the other a representation model.
One attention head can only learn one kind of relationship. So we run many in parallel.
These specializations emerge during training — nobody assigns them.
After N transformer blocks, the final hidden state of the last token gets projected back to the vocabulary.
token logit ───────────────── "on" 8.21 "down" 6.93 "in" 5.74 "upon" 5.10 "quietly" 4.88 … …
token P ───────────────── "on" 0.42 "down" 0.21 "in" 0.11 "upon" 0.07 "quietly" 0.05 … … (sums to 1.0)
"The cat sat" → "on" "The cat sat on" → "the" "The cat sat on the" → "mat" "The cat sat on the mat"→ "." (stop)
Each generated token is appended to the input and the whole pipeline runs again. That's it. That's an LLM.
Every layer in the LLM is a small neural network — dots = neurons, lines = learned weights. The structure varies (linear, with-activation, parallel branches), but the substrate is identical.
Everything we've covered so far produces a base model — a giant next-token predictor. It knows facts and language but doesn't know how to talk. Three more training stages turn it into ChatGPT / Claude / Llama-Chat.
Data: trillions of tokens from web, books, code, papers.
Objective: predict the next token. Nothing else.
Result: knows facts & language but continues text — doesn't follow instructions.
Prompt: "What is 2+2?" Output: "What is 2+2? A) 3 B) 4 C) 5 Answer key: B Question 2: What is …"
It mimics text it's seen — like a textbook, not a helper.
Data: thousands to millions of curated (instruction, response) pairs written by humans.
Objective: same next-token loss, but on chat-formatted conversations with special tokens.
Result: learns the chat format, follows instructions, knows when to stop.
Prompt: "What is 2+2?" Output: "2 + 2 equals 4."
Same model weights, mostly intact — SFT just nudges it toward helpful conversational behavior.
Data: humans rank pairs of model responses ("A is better than B").
Objective: push the model toward responses humans prefer (helpful, honest, harmless).
Result: politer, safer, refuses bad requests, admits uncertainty, follows tone.
Prompt: "How do I hack my
neighbor's wifi?"
Output: "I can't help with
unauthorized access. If it's
your own network, here's
how to recover credentials…"
RLHF shapes style and values, not raw knowledge.
~99% of compute goes into pre-training (the architecture from this whole talk). The last two stages are tiny by comparison — but they're what makes the difference between a "fancy autocomplete" and something you can actually have a conversation with.
A multi-turn conversation is just repeated next-token prediction over a growing prompt — wrapped in special tokens, streamed back one token at a time.
There's no hidden state between turns. The entire conversation history is concatenated and re-sent on every turn:
turn 1 prompt: <sys>… <user>Hi</> <asst> turn 2 prompt: <sys>… <user>Hi</> <asst>Hello!</> <user>How are you?</> <asst> turn 3 prompt: <sys>… <user>Hi</> <asst>Hello!</> <user>How are you?</> <asst>Doing well!</> <user>What did I say first?</> <asst>
This is why long conversations get expensive — the prompt grows unboundedly until it hits the context window limit.
Naïve approach: re-run the full forward pass for the whole prompt on every new token. Cost: O(N²) per token, O(N³) per response.
Real approach: cache K and V for every past token. New token only needs to compute its own Q, then attend to the cached K/V.
per-token cost: without KV cache → O(N²) with KV cache → O(N)
The KV cache is why first token is slow ("prefill") but subsequent tokens are fast ("decode"). It also dominates GPU memory in long conversations.
A chatbot answers in one shot. An agent is the same model wrapped in a loop, given tools to take actions in the world and memory to remember things across turns.
Tools are added declaratively in the system prompt as JSON schemas. The model is fine-tuned (during SFT) to emit special tokens that signal a tool call.
<|sys|>You are a helpful assistant.<|/> <|user|>What's the weather in SF?<|/> <|asst|>
<|sys|>You are a helpful assistant.
You have access to the following tools:
[
{
"name": "get_weather",
"description": "Get current weather for a city",
"parameters": {
"city": {"type": "string", "required": true}
}
},
{ "name": "search_web", … },
{ "name": "send_email", … }
]<|/>
<|user|>What's the weather in SF?<|/>
<|asst|>
<|tool_call|> {"name": "get_weather", "args": {"city": "SF"}} <|/tool_call|>
<|tool_result|> {"temp_f": 62, "conditions": "foggy"} <|/tool_result|> <|asst|>It's 62°F and foggy in SF right now.<|/>
The model never actually called an API — it just emitted text. The agent runtime parses the tool-call tokens, executes the real function, and stuffs the result back into the prompt. Same loop the chatbot uses, just with extra structured turns.
The model itself is stateless — it forgets everything between API calls. "Memory" is built around the model by reading/writing to external stores and stuffing relevant bits back into the prompt.
Just keep appending turns until the context window fills up. Free, automatic, but bounded.
When history grows too long, ask the model to summarize old turns and replace them with the summary. Lossy but cheap.
Embed past conversations (or external docs) into vectors. At each turn, retrieve the top-k most similar chunks and inject them. Scales to millions of facts.
Give the agent read_memory / write_memory tools. The model decides what to save and when to recall — explicit and durable across sessions.
<|sys|>You are a helpful assistant.
## Recalled memory (relevant facts):
- User prefers metric units
- User is based in Berlin
- Last conversation: discussed Python tooling
[tool definitions…]<|/>
<|user|>What's the weather?<|/>
Whether memory comes from a summary, a vector DB, or a file — it always ends up the same way: extra text in the prompt. The model sees no difference.
Tools and memory don't change the model. They change what's in the prompt at each step. An "agent framework" (LangChain, AutoGPT, Claude's tool use API, etc.) is essentially a prompt-building loop that decides what to put in front of the LLM next based on what it just emitted.
Questions?