The Fundamentals of Training an LLM

You use ChatGPT, Claude, and Copilot every day. But if someone asked you how training actually works, could you explain it?

Most developers can't. And most of what's available to learn from doesn't help. Academic papers assume a graduate-level math background. Blog posts stop at "transformers use attention mechanisms" and call it an explanation. The middle ground — deep enough to build real understanding, practical enough to produce working code — barely exists.

This book occupies that middle ground.

Over sixteen chapters, you'll build a complete GPT-style language model from scratch in Python and PyTorch — approximately 800,000 parameters, architecturally identical to production models, small enough to train on your laptop. Then you'll fine-tune a real pre-trained model using LoRA and the Hugging Face ecosystem. Every concept is explained through code, not equations. Every chapter produces something that runs.

What you'll learn:

• How LLMs actually work — tokens, embeddings, attention, and the transformer architecture built from the ground up
• Loss functions and cross-entropy — what "wrong" means mathematically, and how minimizing a single number teaches a model language
• Gradient descent and backpropagation — the algorithm that makes learning possible, implemented step by step
• Tokenization — build a character-level tokenizer and a BPE tokenizer from scratch
• The attention mechanism — query, key, value matrices, scaled dot-product attention, causal masking, and multi-head attention, explained through working code
• The complete transformer block — attention, feed-forward networks, layer normalization, and residual connections
• Training your model — datasets, data loaders, training loops, learning rate scheduling, and monitoring
• Fine-tuning with LoRA — adapt a real pre-trained model to your own data using parameter-efficient techniques
• Training at scale — mixed precision, gradient accumulation, QLoRA, DeepSpeed, and the broader ecosystem

Every code example is available as a runnable script in the companion GitHub repository. Clone it, run it, modify it, break it, learn from it.

This book is for you if you're a Python developer who uses AI tools daily and wants to understand what's actually happening beneath the API, you learn best by writing and running code, or you're ready to go deeper than blog posts without wading through academic papers.

Prerequisites: Python proficiency and basic familiarity with NumPy. No machine learning experience required — this book starts from first principles and builds up.

Book 1 in The Agent Stack: LLMs, Agents, and Multi-Agent Systems. Companion repository at github.com/grizzlypeaksoftware/gps-llm-training-fundamentals.