#5757th of 118 in Learning and cookbooks

how-to-train-your-gpt

Twelve-chapter PyTorch course on building a LLaMA-style GPT from scratch

Stars
4.2k
License
MIT
Last commit
Oct 2026
Language
Jupyter Notebook
Stars, last 27 weeks+3,385
9 May 2026: 797 stars16 May 2026: 1,571 stars23 May 2026: 1,735 stars30 May 2026: 2,127 stars6 Jun 2026: 2,181 stars13 Jun 2026: 2,203 stars20 Jun 2026: 2,217 stars27 Jun 2026: 2,233 stars4 Jul 2026: 2,249 stars11 Jul 2026: 2,260 stars18 Jul 2026: 2,270 stars25 Jul 2026: 2,279 stars1 Aug 2026: 2,286 stars8 Aug 2026: 2,301 stars15 Aug 2026: 2,325 stars22 Aug 2026: 2,799 stars29 Aug 2026: 3,199 stars5 Sept 2026: 3,264 stars12 Sept 2026: 3,311 stars19 Sept 2026: 3,331 stars26 Sept 2026: 3,348 stars3 Oct 2026: 3,616 stars10 Oct 2026: 3,641 stars17 Oct 2026: 3,935 stars24 Oct 2026: 4,130 stars31 Oct 2026: 4,156 stars7 Nov 2026: 4,174 stars14 Nov 2026: 4,182 stars

797 on 9 May 20264,182 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

A written course with companion Jupyter notebooks that walks through building a decoder-only transformer in PyTorch: BPE tokenizer, RoPE, attention, RMSNorm, SwiGLU, training loop and inference with KV cache. A runnable main.py trains a 17M-parameter config on CPU, with a 151M-parameter GPT-2-scale config that needs a GPU. It also includes 28 topic explainers and a short LoRA/QLoRA fine-tuning guide.

Who it is for: Python developers who want to understand transformers by building one

Strengths

  • Every line of code is annotated; chapters pair analogies with worked numeric examples
  • Covers modern components: RoPE, RMSNorm, SwiGLU, KV cache, top-k/top-p, beam search
  • Default 17M-parameter config trains on CPU in minutes; Colab notebook provided
  • Notebooks per chapter plus a LoRA/QLoRA fine-tuning section

Weaknesses

  • Stated as learning-only; not a training framework for real models
  • Explainer folder is named 'WIP', so some content may be unfinished
  • 151M-parameter config requires manual edits to main.py and a GPU
  • Author says AI was used to understand and verify concepts; accuracy not independently reviewed

What it needs

  • GPU optional
  • Needs PyTorch, tiktoken, datasets, numpy, matplotlib, Jupyter
  • Models: LLaMA 3-style decoder-only transformer (17M and 151M configs)

Also in Learning and cookbooks

See all 118
Also in Learning and cookbooks
RankProjectScore
1ai-engineering-from-scratchFree curriculum of 542 coding lessons on building AI systems from scratch79.4k starsMIT91 out of 100
2generative-ai-for-beginners21-lesson course on building generative AI apps in Python and TypeScript123.8k starsMIT83 out of 100
3rag_techniquesCollection of 42 runnable notebooks implementing RAG techniques30.4k starscustom license83 out of 100
#4cs249r_bookOpen textbook series on ML systems, with TinyTorch, labs and hardware kits30.2k starscustom license83 out of 100
#5agents-towards-productionJupyter tutorials for taking LLM agents from prototype to production21.9k starscustom license82 out of 100