#5757th of 118 in Learning and cookbooks
how-to-train-your-gpt
Twelve-chapter PyTorch course on building a LLaMA-style GPT from scratch
- Stars
- 4.2k
- License
- MIT
- Last commit
- Oct 2026
- Language
- Jupyter Notebook
797 on 9 May 20264,182 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
A written course with companion Jupyter notebooks that walks through building a decoder-only transformer in PyTorch: BPE tokenizer, RoPE, attention, RMSNorm, SwiGLU, training loop and inference with KV cache. A runnable main.py trains a 17M-parameter config on CPU, with a 151M-parameter GPT-2-scale config that needs a GPU. It also includes 28 topic explainers and a short LoRA/QLoRA fine-tuning guide.
Who it is for: Python developers who want to understand transformers by building one
Strengths
- Every line of code is annotated; chapters pair analogies with worked numeric examples
- Covers modern components: RoPE, RMSNorm, SwiGLU, KV cache, top-k/top-p, beam search
- Default 17M-parameter config trains on CPU in minutes; Colab notebook provided
- Notebooks per chapter plus a LoRA/QLoRA fine-tuning section
Weaknesses
- Stated as learning-only; not a training framework for real models
- Explainer folder is named 'WIP', so some content may be unfinished
- 151M-parameter config requires manual edits to main.py and a GPU
- Author says AI was used to understand and verify concepts; accuracy not independently reviewed
What it needs
- GPU optional
- Needs PyTorch, tiktoken, datasets, numpy, matplotlib, Jupyter
- Models: LLaMA 3-style decoder-only transformer (17M and 151M configs)
Also in Learning and cookbooks
See all 118| Rank | Project | Score |
|---|---|---|
| 1 | ai-engineering-from-scratchFree curriculum of 542 coding lessons on building AI systems from scratch | 91 out of 100 |
| 2 | generative-ai-for-beginners21-lesson course on building generative AI apps in Python and TypeScript | 83 out of 100 |
| 3 | rag_techniquesCollection of 42 runnable notebooks implementing RAG techniques | 83 out of 100 |
| #4 | cs249r_bookOpen textbook series on ML systems, with TinyTorch, labs and hardware kits | 83 out of 100 |
| #5 | agents-towards-productionJupyter tutorials for taking LLM agents from prototype to production | 82 out of 100 |