#2323rd of 65 in Learning and cookbooks
train-llm-from-scratch
Hand-written PyTorch pipeline from raw text to a pretrained, aligned LLM
- Stars
- 13.1k
- License
- MIT
- Last commit
- Oct 2026
- Language
- Python
536 on 4 Apr 202613,077 on 31 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Tutorial repo that implements a transformer in plain PyTorch and trains it from raw text through SFT, reward model, PPO, DPO and GRPO, without trl, peft or transformers. It has a classic GPT-style model and a modern one (RoPE, RMSNorm, SwiGLU, GQA/MLA, MoE, KV cache), plus a 369K-parameter preset that trains on a laptop CPU. A Streamlit control panel and an MkDocs site are included.
Who it is for: Students and engineers learning how LLMs are pretrained and aligned
Strengths
- Pretraining and post-training (SFT, reward, PPO, DPO/ORPO/KTO, GRPO) all hand-written in PyTorch
- Tiny preset trains on a laptop CPU in about five minutes
- Typed JSON configs per stage, smoke configs that finish in seconds, and pytest coverage
- Modern model option with Muon, LoRA, speculative decoding and int8 weights
Weaknesses
- Educational scale: the sample output from the 13M model is largely incoherent
- Billion-parameter training needs a 24 GB+ GPU; larger configs are marked TBD on RTX 5090
- Two config systems (legacy constants and JSON) add confusion
- No release tags and no Docker setup
What it needs
- GPU optional
- Needs PyTorch, tiktoken, Hugging Face datasets, Streamlit, wandb, HDF5
- Models: Custom transformer (classic and modern), PyTorch checkpoints
Quick answers
Is train-llm-from-scratch open source?
Yes. Its code is on GitHub under the MIT license.
Does train-llm-from-scratch need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Learning and cookbooks
See all 65| Rank | Project | Score |
|---|---|---|
| 1 | ai-engineering-from-scratchHands-on AI engineering curriculum with 542 lessons across 20 phases | 89 out of 100 |
| 2 | generative-ai-for-beginners21-lesson course on building generative AI apps in Python and TypeScript | 81 out of 100 |
| 3 | rag_techniquesCollection of 42 runnable notebooks implementing RAG techniques | 79 out of 100 |
| #4 | cs249r_bookOpen textbook and course materials for machine learning systems engineering | 79 out of 100 |
| #5 | ai-agents-for-beginners18-lesson course on building AI agents with Microsoft Agent Framework | 78 out of 100 |