#44th of 5 in Training and fine-tuning
nanochat
Minimal PyTorch codebase to train and chat with a small GPT-2-class LLM
- Stars
- 58.6k
- License
- MIT
- Last commit
- Jul 2026
- Language
- Python
49,466 on 24 Mar 202658,621 on 13 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
nanochat covers tokenizer training, pretraining, SFT, RL, evaluation and a CLI chat interface in one small Python codebase. A single `--depth` setting derives width, heads, learning rate and training horizon, so one knob produces a compute-optimal model series. The reference speedrun trains a GPT-2-grade model in about 1.65 hours on an 8xH100 node per the leaderboard.
Who it is for: Researchers and engineers learning or experimenting with end-to-end LLM training
Strengths
- Covers tokenizer, pretraining, SFT, RL, eval and inference in one repo
- Single `--depth` dial sets all other hyperparameters automatically
- Public time-to-GPT-2 leaderboard with commits and CORE scores
- Same scripts run on one GPU via gradient accumulation, or on CPU/MPS
Weaknesses
- Reference speedrun needs an 8xH100 node, roughly $48 of rented compute
- Under 80GB VRAM requires manually lowering `--device-batch-size`
- CPU/MPS path only trains a tiny model with weak results
- Chat is CLI-only in the README; resulting model is small and hallucinates
What it needs
- GPU required
- Needs PyTorch, uv, wandb
- Models: nanochat GPT (depth-scaled), bfloat16, float32, float16
Head to head
Also in Training and fine-tuning
See all 5| Rank | Project | Score |
|---|---|---|
| 1 | llamafactoryFine-tuning toolkit for 100+ LLMs with a CLI and Gradio web UI | 74 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 72 out of 100 |
| 3 | minimindTrain a 64M-parameter LLM from scratch in plain PyTorch | 64 out of 100 |
| #5 | colossalaiDistributed training library for large models with many parallelism strategies | 43 out of 100 |