33rd of 8 in Learning and cookbooks
llms-from-scratch
Code for the book on building a GPT-style LLM in PyTorch
- Stars
- 106.5k
- License
- custom license
- Last commit
- Oct 2026
- Language
- Jupyter Notebook
Checked on GitHub.
Overview
Official code for the Manning book "Build a Large Language Model (From Scratch)". Notebooks and scripts implement tokenization, attention, a GPT model, pretraining, classification finetuning, instruction finetuning and LoRA in plain PyTorch, without external LLM libraries. Bonus folders add from-scratch Llama 3.2, Qwen3, Gemma 3, Olmo 3, KV cache, MoE and DPO.
Who it is for: Python developers learning how LLMs work by coding one in PyTorch
Strengths
- Main chapter code runs on a conventional laptop and uses a GPU automatically if present
- Plain PyTorch with no external LLM libraries, so every component is readable
- Bonus material covers KV cache, GQA, MoE, DPO and several model architectures
- Exercise solutions, a companion video course and CI tests on Linux, Windows and macOS
Weaknesses
- Tied to a print book; contributions extending main chapter code are not accepted
- Produces a small educational model, not a production-grade LLM
- License is reported as NOASSERTION; the README states no license terms
- Video course and quiz PDF are hosted by the publisher, not in the repo
What it needs
- GPU optional
- Needs PyTorch, Jupyter, Ollama (optional, ch07 evaluation)
- Models: GPT-2, Llama 3.2, Qwen3, Gemma 3, Olmo 3
Head to head
Also in Learning and cookbooks
See all 8| Rank | Project | Score |
|---|---|---|
| 1 | generative-ai-for-beginners21-lesson course on building generative AI apps in Python and TypeScript | 80 out of 100 |
| 2 | ai-engineering-from-scratchHands-on curriculum for building AI systems from math to agents | 75 out of 100 |
| #4 | hello-agentsChinese-language course on building LLM agents from scratch | 72 out of 100 |
| #5 | ai-agents-for-beginners18-lesson course on building AI agents with Microsoft Agent Framework | 70 out of 100 |
| #6 | openai-cookbookExample notebooks and guides for building with the OpenAI API | 65 out of 100 |