← The decoderhow the model you train here runs, one forward pass
Transformer Training Explained
Training the decoder
An interactive guide to how the Decoder site’s model is trained, from the loss and backprop to many GPUs, with the model training in your browser.
How this is checkedA NumPy reference of the trainer matches PyTorch autograd (to 1e-12) and torch.optim, and the trainer in your browser matches that reference step for step. How it is tested →
What you’ll learn
The Decoder site runs a small decoder with random weights. This site trains it: the same model, from the same initial weights, on a public-domain book, in your browser. Chapter 1 takes it from a validation loss of 4.18 (random guessing over 64 characters) to 2.30 in 2,000 steps. Every number on these pages comes from a recorded, seeded run (results.md), and the in-browser trainer is checked step by step against a reference whose gradients match PyTorch’s.
New to the model itself? Start with the Decoder site. Prefer slides? Every chapter has a deck in the Transformer Training deck series.
Batch, forward, loss, backward, update: the Decoder's model trains in your browser.
A public-domain corpus, BPE merges learnt live, windows and the validation split.
Next-token cross-entropy, teacher forcing, perplexity, and the gradient p − y.
The gradient's way back, op by op, and what the forward pass must save.
SGD, momentum, Adam and AdamW, on a valley and on the real model.
Warmup, cosine and WSD; the learning-rate sweep; batch size and gradient noise.
Pre-LN vs post-LN, RMSNorm, scale through depth, hyperparameter transfer.
A loss spike reproduced and cured: clipping, z-loss, QK-norm.
FP32 master weights, bf16 and fp16 with loss scaling, FP8, quantised gradients.
- 10Memory
Parameters, gradients, optimiser states and activations; checkpointing.
Data parallelism, all-reduce, reduce-scatter and all-gather, ZeRO and FSDP.
Tensor, pipeline, sequence and expert parallelism; the pipeline bubble.
Top-k with error feedback, DeMo and DiLoCo when workers share a WAN.
Kaplan and Chinchilla, IsoFLOP curves, compute-optimal training.
Throughput, MFU, monitoring, failures, checkpoints and goodput.
Mid-training, SFT, preference optimisation and RL, in outline.