PyTorch 101

Creating Tensors

import torch

# From a Python list
x = torch.tensor([1, 2, 3])         # 1-D, shape (3,)
x = torch.tensor([[1, 2], [3, 4]])  # 2-D, shape (2, 2)

# Special constructors
torch.zeros(2, 3)                    # 2x3 of zeros
torch.ones(5)                        # length-5 of ones
torch.arange(10)                     # [0,1,…,9]
torch.randn(4, 8)                    # Gaussian noise, shape (4,8)
torch.randint(0, 100, (2, 5))        # random ints in [0,100)

Inspecting Tensors

x.shape         # torch.Size([2, 3])
x.dtype         # torch.float32
x.device        # device(type='cpu')
x.numel()       # total number of elements

Visualizing Shape

A tensor of shape (2, 3):

A 4-D tensor in deep learning is typically (batch, time, heads, dim). You'll see this constantly in attention.

A Tiny Example

# Two learnable scalars
w = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)
x = torch.tensor([3.0])              # input (no grad)

# Forward pass — every op gets recorded
a = w * x         # 6.0   grad_fn=<MulBackward>
z = a + b         # 7.0   grad_fn=<AddBackward>
y = z ** 2        # 49.0  grad_fn=<PowBackward>

print(y.grad_fn)
# <PowBackward0 object at 0x…>

The model never knows what y "is." It only knows how to undo the last op (grad_fn) and which tensors to ask next.

Three Kinds of Tensors in the Graph

Orange-rimmed = leaf with requires_grad. Purple = op node (knows how to compute its derivative). Green = output we'll backprop from.

Local Derivative for Each Op

Each op type knows its own forward formula and its own derivative. PyTorch ships these as built-in Backward classes.

Chain Them at Each Leaf

# Seed: ∂y/∂y = 1

∂y/∂w = ∂y/∂z · ∂z/∂a · ∂a/∂w
      =   14   ·   1   ·   3     = 42

∂y/∂b = ∂y/∂z · ∂z/∂b
      =   14   ·   1            = 14

PyTorch does exactly this — but for millions of parameters at once, with matrix-valued local derivatives (Jacobians).

The Code That Triggers All This

w = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([1.0], requires_grad=True)
x = torch.tensor([3.0])

y = (w * x + b) ** 2      # 49.0
y.backward()                # ← runs the whole reverse pass

print(w.grad)              # tensor([42.])
print(b.grad)              # tensor([14.])

The Pattern

import torch.nn as nn
import torch.nn.functional as F

class FeedForward(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        # declare learnable sub-modules here
        self.fc1 = nn.Linear(d_model, 4 * d_model)
        self.fc2 = nn.Linear(4 * d_model, d_model)

    def forward(self, x):
        # describe the computation
        return self.fc2(F.gelu(self.fc1(x)))

# use it
ff = FeedForward(128)
y  = ff(x)        # calls forward() under the hood

What Module Does For You

• Auto-registers sub-modules and parameters — anything assigned to self.X that's a Module or Parameter.
• model.parameters() returns an iterator over all learnable tensors recursively. The optimizer uses this.
• model.to(device) moves everything at once.
• model.state_dict() dumps all weights for saving.
• model.train() / model.eval() toggles modes (dropout/batchnorm care).

Side by Side

Two ways to apply GeLU. Both produce the same output:

class FFN(nn.Module):
    def __init__(self):
        super().__init__()
        self.act = nn.GELU()
        self.fc  = nn.Linear(128, 128)

    def forward(self, x):
        return self.fc(self.act(x))

class FFN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(128, 128)

    def forward(self, x):
        return self.fc(F.gelu(x))   # ← cleaner

The Functional Ops You'll Use

Shape Manipulation

x = torch.arange(6)              # tensor([0,1,2,3,4,5])  shape (6,)
x.view(2, 3)
# tensor([[0, 1, 2],
#         [3, 4, 5]])    shape (2, 3)

x = torch.tensor([[1,2,3],[4,5,6]])   # (2, 3)
x.transpose(0, 1)
# tensor([[1, 4],
#         [2, 5],
#         [3, 6]])       shape (3, 2)

x = torch.randn(2, 3, 4)         # shape (2, 3, 4)
x.permute(2, 0, 1).shape       # torch.Size([4, 2, 3])

x = torch.tensor([1, 2, 3])      # shape (3,)
x.unsqueeze(0).shape           # torch.Size([1, 3])
x.unsqueeze(1).shape           # torch.Size([3, 1])

y = torch.zeros(1, 3, 1)
y.squeeze().shape              # torch.Size([3])

x = torch.arange(12).view(3, 4)
a, b = x.chunk(2, dim=1)        # split last dim in 2
# a, b each have shape (3, 2)
# Used in attention: q, k, v = qkv.chunk(3, dim=-1)

a = torch.tensor([1, 2, 3])      # shape (3,)
b = torch.tensor([4, 5, 6])      # shape (3,)

torch.cat([a, b], dim=0)
# tensor([1,2,3,4,5,6])     shape (6,)   — joins along existing dim

torch.stack([a, b], dim=0)
# tensor([[1,2,3],
#         [4,5,6]])         shape (2,3)  — creates a NEW dim

x = torch.randn(2, 3).transpose(0, 1)
# transpose doesn't move data — just changes strides
x.view(6)                       # ❌ RuntimeError: view size not compatible
x.contiguous().view(6)          # ✅ now it's a flat copy

Math & Reductions

a = torch.tensor([1, 2, 3])
b = torch.tensor([10, 20, 30])

a + b      # tensor([11, 22, 33])
a * b      # tensor([10, 40, 90])
b / a      # tensor([10., 10., 10.])
a + 100    # tensor([101, 102, 103])  — scalar broadcasts

A = torch.tensor([[1., 2.],
                  [3., 4.]])             # (2, 2)
B = torch.tensor([[5., 6.],
                  [7., 8.]])             # (2, 2)

A @ B
# tensor([[19., 22.],
#         [43., 50.]])      shape (2, 2)

# Batched: works for any leading dims
x = torch.randn(8, 4, 3, 5)         # (8, 4, 3, 5)
y = torch.randn(8, 4, 5, 7)         # (8, 4, 5, 7)
(x @ y).shape                    # torch.Size([8, 4, 3, 7])

x = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.]])             # (2, 3)

x.sum()                       # tensor(21.)         scalar
x.sum(dim=0)                  # tensor([5., 7., 9.])  shape (3,)
x.sum(dim=1)                  # tensor([6., 15.])     shape (2,)
x.sum(dim=1, keepdim=True)    # shape (2, 1)  — useful for broadcasting

x.mean(dim=-1)                # tensor([2., 5.])
x.max()                       # tensor(6.)

logits = torch.tensor([1.2, 3.4, 0.8, 5.6, 2.1])

logits.argmax()                # tensor(3)   — index of max

vals, idx = torch.topk(logits, k=2)
# vals = tensor([5.6, 3.4])
# idx  = tensor([3, 1])      ← used in top-k sampling

Indexing & Masking

x = torch.arange(24).view(2, 3, 4)   # shape (2, 3, 4)

x[0]            # first batch:           shape (3, 4)
x[:, -1, :]     # last token, all batch: shape (2, 4)
x[..., 0]       # first elem of last dim: shape (2, 3)
x[:, :, :2]     # first 2 along last dim: shape (2, 3, 2)

x = torch.tensor([10, 20, 30, 40, 50])
idx = torch.tensor([0, 2, 4])

x[idx]
# tensor([10, 30, 50])

# This is exactly how nn.Embedding works internally

x = torch.tensor([1., 2., 3., 4.])
mask = torch.tensor([True, False, True, False])

x.masked_fill(mask, float('-inf'))
# tensor([-inf, 2., -inf, 4.])

# Causal mask in attention:
mask = torch.triu(torch.ones(T,T), diagonal=1).bool()
scores.masked_fill(mask, float('-inf'))   # future → -inf → 0 after softmax

Sampling & Random

torch.randint(0, 100, (2, 3))
# tensor([[42,  7, 91],
#         [ 3, 55, 28]])   shape (2, 3)  — used in get_batch()

torch.randn(3)
# tensor([0.21, -1.45, 0.83])  — mean 0, std 1 (used to init weights)

probs = torch.tensor([0.1, 0.7, 0.2])
torch.multinomial(probs, num_samples=1)
# tensor([1])    — most likely 1 (highest prob), occasionally 0 or 2
# Used in generate() to sample the next token

torch.arange(4)               # tensor([0, 1, 2, 3])  — position indices

torch.tril(torch.ones(4, 4))
# tensor([[1, 0, 0, 0],
#         [1, 1, 0, 0],
#         [1, 1, 1, 0],
#         [1, 1, 1, 1]])    ← causal mask in attention

Worked Examples

x    = torch.randn(32, 128)        # (B, D)
bias = torch.randn(128)             # (D,)

x + bias
# bias is treated as (1, 128) → expanded to (32, 128)
# Same vector added to every row

col = torch.tensor([[1], [2], [3]])    # shape (3, 1)
row = torch.tensor([[10, 20, 30, 40]]) # shape (1, 4)

col + row
# tensor([[11, 21, 31, 41],
#         [12, 22, 32, 42],
#         [13, 23, 33, 43]])    shape (3, 4)

tok = self.tok_embed(idx)        # (B, T, D)
pos = self.pos_embed(torch.arange(T))   # (T, D)

x = tok + pos
# pos is treated as (1, T, D) → expanded to (B, T, D)
# Same position vectors added to every batch item

a = torch.randn(3, 4)
b = torch.randn(3, 5)
a + b   # RuntimeError: 4 ≠ 5 and neither is 1

Common Optimizers — When to Use What

Quick Code

torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
torch.optim.Adam(model.parameters(), lr=3e-4)
torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
                                          # LLM default

Rules of thumb: training a transformer → AdamW · training a CNN → SGD+momentum · unsure → AdamW with lr=3e-4 ("the Karpathy constant") · out of GPU memory → switch to Adafactor or 8-bit Adam (bitsandbytes).

Pair any of these with a learning-rate scheduler (cosine decay, warmup) — the schedule often matters more than the optimizer choice.

Order Matters — Why?

Mnemonic: Z·F·B·S — Zero, Forward, Backward, Step.

Bonus: Gradient Accumulation

Want a bigger effective batch than fits in memory? Skip zero_grad and step for K steps:

for i, (x, y) in enumerate(dataloader):
    loss = model(x, y) / ACCUM_STEPS
    loss.backward()                # grads keep accumulating

    if (i + 1) % ACCUM_STEPS == 0:   # every K steps:
        optimizer.step()             #   apply combined grads
        optimizer.zero_grad()        #   then clear

Same math as a K× larger batch — but with K× less peak memory. The "accumulation" you usually fight is now your friend.

Two Things to Toggle

# As a context manager
with torch.no_grad():
    logits = model(x)
    preds  = logits.argmax(dim=-1)

# Or as a decorator (cleaner for whole functions)
@torch.no_grad()
def generate(model, idx, max_new_tokens):
    for _ in range(max_new_tokens):
        ...
        idx = torch.cat([idx, next_token], dim=1)
    return idx

model.eval()      # dropout off, batchnorm uses running stats
# …generate / predict…
model.train()     # back to training mode

Why Bother?

• Memory: autograd stores intermediate activations for the backward pass. no_grad skips this — often 2-4× less memory.
• Speed: no graph construction, no .grad bookkeeping → typically 10-30% faster.
• Correctness: Dropout layers behave wildly differently in train vs eval mode. Forgetting model.eval() is a classic bug.

The Three-Step Inference Recipe

model.eval()
with torch.no_grad():
    output = model(x)

Run this anytime you're predicting/generating, not training.

Saving

# Get a dict of {param_name → tensor}
state = model.state_dict()

# Save it (plus anything else you'll need later)
torch.save({
    "model":     state,
    "optimizer": optimizer.state_dict(),
    "config":    config,
    "step":      step,
}, "checkpoints/model.pt")

Loading

ckpt = torch.load(
    "checkpoints/model.pt",
    map_location=device,        # load onto current device
    weights_only=False,        # allow non-tensor objects (config)
)

# Rebuild the architecture from code
model = GPT(ckpt["config"]).to(device)
model.load_state_dict(ckpt["model"])

What's a state_dict?

An ordered dict mapping every parameter (and buffer) to its tensor. For our tiny GPT it looks roughly like:

{
  "tok_embed.weight":        tensor(50257, 128),
  "pos_embed.weight":        tensor(128, 128),
  "blocks.0.ln1.weight":     tensor(128),
  "blocks.0.ln1.bias":       tensor(128),
  "blocks.0.attn.qkv.weight": tensor(384, 128),
  "blocks.0.attn.proj.weight": tensor(128, 128),
  "blocks.0.ffn.fc1.weight":  tensor(512, 128),
  "blocks.0.ffn.fc1.bias":    tensor(512),
  ...
}

Picking the Best Available Device

device = (
    "cuda"  if torch.cuda.is_available()
    else "mps"   if torch.backends.mps.is_available()
    else "cpu"
)
print(f"running on {device}")

Moving Things Around

# Move the whole model
model = model.to(device)

# Move tensors
x = x.to(device)
y = y.to(device)

# Or create directly on a device
x = torch.zeros(3, 4, device=device)

Speed Comparison (rough)

For training large models, GPUs aren't a luxury — they're a necessity.

Bonus: map_location

ckpt = torch.load("model.pt", map_location=device)

Load a checkpoint that was saved on a CUDA machine onto a CPU/MPS machine. Without this you'll hit a "no CUDA" error.

What is GGUF?

GGUF = GPT-Generated Unified Format. A binary container designed for fast, memory-mapped inference. Built by Georgi Gerganov for llama.cpp; now the de-facto standard for local LLM deployment.

What's Inside

• Model weights — usually quantized to 2–8 bits
• Tokenizer — vocab + merges, embedded in the file
• Architecture metadata — layer count, head count, context length
• Chat template — how to wrap conversations

One file, no dependencies. Memory-mapped → loads instantly even for 70B models. Replaces the older GGML format.

Quantization Levels — Pick Your Trade-off

A 7B Llama at Q4_K_M weighs ~4 GB and runs comfortably on a 16 GB laptop.

The Ollama Modelfile

Like a Dockerfile for AI models. Declares: base weights, system prompt, sampling defaults, chat template. Build it once → run it anywhere Ollama runs.

Basic Modelfile Syntax

FROM      <base>           # required: model to start from
SYSTEM    "..."            # default system prompt
PARAMETER temperature 0.7  # sampling knobs
PARAMETER top_p       0.9
PARAMETER num_ctx     8192
TEMPLATE  "..."            # optional: chat format override
LICENSE   "..."            # optional: license text

Example: A Custom Python Tutor

# Modelfile
FROM llama3.2:3b

SYSTEM """You are a patient Python tutor.
Always show runnable code examples and
explain edge cases."""

PARAMETER temperature 0.5
PARAMETER top_p       0.9
PARAMETER num_ctx     8192
PARAMETER stop        "<|eot_id|>"

Build & Run

$ ollama create py-tutor -f Modelfile
# transferring model data 100%
# creating model layer
# writing manifest
# success

$ ollama list
# NAME              ID            SIZE     MODIFIED
# py-tutor:latest   a3b1c…        2.0 GB   3 minutes ago
# llama3.2:3b       7d8c9…        2.0 GB   2 days ago

$ ollama show py-tutor              # config, params, template
$ ollama show py-tutor --modelfile  # dump the Modelfile
$ ollama cp py-tutor py-tutor-v2    # clone
$ ollama rm py-tutor                # delete
$ ollama pull llama3.2:3b           # fetch from registry
$ ollama push user/py-tutor         # publish to ollama.com

$ ollama run py-tutor
>>> Explain decorators with an example.
# …streaming response…
>>> /set system "Now answer in haikus."   # override at runtime
>>> /show modelfile                          # inspect
>>> /bye                                     # exit

$ ollama run py-tutor "Why is Python slow?"

$ echo "Summarize this:" | ollama run py-tutor
cat log.txt | ollama run py-tutor "Find errors:"

$ ollama serve              # starts on http://localhost:11434

# Generate (single-turn)
$ curl http://localhost:11434/api/generate -d '{
    "model": "py-tutor",
    "prompt": "Explain async/await",
    "stream": false
  }'

# Chat (multi-turn, OpenAI-style messages)
$ curl http://localhost:11434/api/chat -d '{
    "model": "py-tutor",
    "messages": [
      {"role": "user", "content": "Hi"}
    ]
  }'

# pip install ollama
from ollama import chat

response = chat(
    model="py-tutor",
    messages=[{"role": "user", "content": "Explain GIL"}],
)
print(response.message.content)

# Or stream tokens as they arrive
for chunk in chat(model="py-tutor", messages=[...], stream=True):
    print(chunk.message.content, end="", flush=True)

Loading Your Own GGUF

# Modelfile
FROM ./my-model.Q4_K_M.gguf
SYSTEM "You are my fine-tuned model."
PARAMETER temperature 0.3

Point FROM at any local .gguf file. Ollama wraps it with your config and the same create / list / run / serve commands above just work.

The Core Concepts

What to Read Next

• Official tutorials: pytorch.org/tutorials — the "Learn the Basics" track
• Karpathy's nanoGPT: the canonical small GPT in PyTorch
• The PyTorch source: reading nn/modules/linear.py teaches you how to write your own layer
• This deck's companion code: tiny_llm/ — every concept here used in real working code

You Now Have Enough To…

• ✓ Read the entire tiny_llm/model.py source
• ✓ Modify the architecture (add layers, change dimensions)
• ✓ Write your own training loop on a different dataset
• ✓ Save and resume training
• ✓ Move work to / from GPU
• ✓ Debug shape and device errors