Build LLMs from scratch.
From backpropagation by hand to transformers, reasoning, agents and MCP. Each step is a small exercise in plain Python and NumPy, solved and tested right in your browser.
15 sections · 2 problems
- 0
NumPy & Math Foundations
0/1Vectorized thinking, broadcasting and numerically stable primitives every later block relies on.
- 1
Backpropagation by Hand
0/1Derive and code gradients yourself, then prove them right with gradient checking.
- 2
Loss Functions
Coming soonMSE, cross-entropy from logits and their gradients.
- 3
Optimizers & Schedules
Coming soonSGD, momentum, Adam/AdamW, warmup + cosine decay, gradient clipping.
- 4
Neural Network Layers
Coming soonEmbedding, LayerNorm, RMSNorm, dropout and weight initialization.
- 5
Language Modeling Basics
Coming soonBigram and MLP language models, temperature, top-k and top-p sampling.
- 6
Tokenization
Coming soonTrain a byte-pair encoding tokenizer, then encode and decode with it.
- 7
Attention & Transformers
Coming soonScaled dot-product attention, causal masking, multi-head, RoPE, GQA, the full block.
- 8
Training LLMs
Coming soonTraining loops, batching, mixed precision ideas and scaling-law arithmetic.
- 9
Efficient Inference
Coming soonKV cache, beam search, speculative decoding and int8 quantization.
- 10
Post-Training & Alignment
Coming soonSFT loss masking, LoRA, reward models, DPO, PPO-clip and GRPO.
- 11
Reasoning
Coming soonSelf-consistency, pass@k, best-of-N with verifiers.
- 12
Retrieval-Augmented Generation
Coming soonEmbeddings, cosine similarity, BM25, chunking and retrieval.
- 13
Agents
Coming soonTool-call parsing, the ReAct loop and agent memory, driven by a scripted fake LLM.
- 14
Model Context Protocol
Coming soonJSON-RPC 2.0, the MCP handshake and a minimal tool server.