deepspeed vs nanochat
Two of the top projects in Training and fine-tuning, side by side: who leads on each score part and who each one suits, then setup, license, activity and what each review found.
deepspeed
PyTorch library for distributed training of large models with ZeRO
nanochat
Minimal PyTorch codebase to train and chat with a small GPT-2-class LLM
Verdict
deepspeed ranks higher, 72 to 49. Of the 4 score parts we measure for both, deepspeed leads on 3, nanochat on 1.
- Adoptionnanochat leads by 18 points, 62 to 44: a wide gap.
- Freshnessdeepspeed leads by 5 points, 100 to 95.
- Maintenancedeepspeed leads by 79 points, 81 to 2: a wide gap.
- Agent-readydeepspeed leads by 30 points, 45 to 15: a wide gap.
Pick deepspeed if you are
ML engineers training or fine-tuning large models on multi-GPU clusters
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers/Accelerate, Lightning, Composer, Determined and MMEngine
Pick nanochat if you are
Researchers and engineers learning or experimenting with end-to-end LLM training
- Covers tokenizer, pretraining, SFT, RL, eval and inference in one repo
- Single `--depth` dial sets all other hyperparameters automatically
| What we compare | deepspeed | nanochat |
|---|---|---|
| Score parts, out of 100 | ||
| Adoption | 44, known | 62, popular |
| Freshness | 100, active | 95, active |
| Maintenance | 81, healthy | 2, weak |
| Easy to run | Not scored for this kind | Not scored for this kind |
| Agent-ready | 45, minimal | 15, none |
| Facts from GitHub and the README | ||
| Stars | 43.2k | 58.6k |
| License | Apache-2.0 (permissive) | MIT (permissive) |
| Last commit | Oct 2026 | Jul 2026 |
| Last release | Sep 2026 | None published |
| Language | Python | Python |
| Docker | Yes | No |
| GPU | Optional | Required |
| arm64 or Apple Silicon | Not stated | Mentioned |
deepspeed
DeepSpeed is a PyTorch optimization library for training large deep learning models across many GPUs. It provides ZeRO memory partitioning with CPU and NVMe offload, 3D parallelism, Ulysses sequence parallelism and MoE support, configured through a JSON file. It integrates with Hugging Face Transformers and Accelerate, Lightning, MosaicML Composer, Determined and MMEngine.
Who it is for: ML engineers training or fine-tuning large models on multi-GPU clusters
Strengths
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers/Accelerate, Lightning, Composer, Determined and MMEngine
- Supports NVIDIA, AMD, Intel Gaudi/XPU/CPU, Huawei Ascend NPU and Tecorigin accelerators
- Installs from PyPI; C++/CUDA ops are JIT-compiled or pre-built
Weaknesses
- Needs PyTorch installed first and a CUDA or ROCm compiler for JIT ops
- Mainly tested on NVIDIA Pascal to Hopper and AMD MI100/MI200
- Windows lacks async IO and GDS support
- Large JSON config surface; ZeRO and offload tuning requires reading the docs
- GPU optional
- Docker
- Needs PyTorch, CUDA or ROCm compiler (nvcc/hipcc), ninja
- Models: PyTorch models, Hugging Face Transformers
nanochat
nanochat covers tokenizer training, pretraining, SFT, RL, evaluation and a CLI chat interface in one small Python codebase. A single `--depth` setting derives width, heads, learning rate and training horizon, so one knob produces a compute-optimal model series. The reference speedrun trains a GPT-2-grade model in about 1.65 hours on an 8xH100 node per the leaderboard.
Who it is for: Researchers and engineers learning or experimenting with end-to-end LLM training
Strengths
- Covers tokenizer, pretraining, SFT, RL, eval and inference in one repo
- Single `--depth` dial sets all other hyperparameters automatically
- Public time-to-GPT-2 leaderboard with commits and CORE scores
- Same scripts run on one GPU via gradient accumulation, or on CPU/MPS
Weaknesses
- Reference speedrun needs an 8xH100 node, roughly $48 of rented compute
- Under 80GB VRAM requires manually lowering `--device-batch-size`
- CPU/MPS path only trains a tiny model with weak results
- Chat is CLI-only in the README; resulting model is small and hallucinates
- GPU required
- Needs PyTorch, uv, wandb
- Models: nanochat GPT (depth-scaled), bfloat16, float32, float16
More in Training and fine-tuning
Building with these?
We help teams pick an open-source AI stack and run it in production. Client work never changes a rank.