deepspeed vs megatron-lm
Two of the top projects in Training and fine-tuning, side by side: who leads on each score part and who each one suits, then setup, license, activity and what each review found.
deepspeed
PyTorch library for distributed training of very large models with ZeRO
megatron-lm
Distributed training library for large transformer models on NVIDIA GPUs
Verdict
deepspeed ranks higher, 85 to 80. Of the 4 score parts we measure for both, deepspeed leads on 2, megatron-lm on 1 and 1 is even.
- Adoptiondeepspeed leads by 26 points, 87 to 61: a wide gap.
- FreshnessEven at 100.
- Maintenancedeepspeed leads by 4 points, 81 to 77: close to even.
- Agent-readymegatron-lm leads by 40 points, 85 to 45: a wide gap.
Pick deepspeed if you are
ML engineers training or fine-tuning large PyTorch models on multi-GPU clusters
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers and Accelerate, Lightning, Composer and Determined
Pick megatron-lm if you are
ML engineers and research teams training large LLMs on GPU clusters
- Supports TP, PP, DP, EP and CP parallelism for multi-node training
- Mixed precision down to FP8 and FP4
| What we compare | deepspeed | megatron-lm |
|---|---|---|
| Score parts, out of 100 | ||
| Adoption | 87, widely used | 61, popular |
| Freshness | 100, active | 100, active |
| Maintenance | 81, healthy | 77, fair |
| Agent-ready | 45, minimal | 85, ready |
| Facts from GitHub and the README | ||
| Stars | 43.3k | 18.3k |
| License | Apache-2.0 (permissive) | custom license (read the license) |
| Last commit | Oct 2026 | Nov 2026 |
| Last release | Oct 2026 | Oct 2026 |
| Language | Python | Python |
| Docker | Yes | Yes |
| GPU | Optional | Required |
| arm64 or Apple Silicon | Not stated | Not stated |
deepspeed
DeepSpeed is a PyTorch library that makes large-model training fit and scale across GPUs using ZeRO, ZeRO-Infinity offload, 3D parallelism, Ulysses sequence parallelism and DeepSpeed-MoE. It is configured through a JSON file and integrates with Hugging Face Transformers, Accelerate, Lightning, MosaicML Composer, Determined and MMEngine. C++/CUDA ops are built just-in-time with ninja by default.
Who it is for: ML engineers training or fine-tuning large PyTorch models on multi-GPU clusters
Strengths
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers and Accelerate, Lightning, Composer and Determined
- Supports NVIDIA, AMD, Intel Gaudi and XPU, Huawei Ascend NPU and CPU accelerators
- Installs with pip install deepspeed; ds_report checks which ops are compatible
Weaknesses
- PyTorch must be installed first, and a CUDA or ROCm compiler is needed for JIT ops
- Windows lacks async IO and GDS support
- Tested mainly on NVIDIA Pascal to Hopper and AMD MI100/MI200; other hardware is less validated
- Training-focused; tuning requires understanding the JSON config and ZeRO stages
- GPU optional
- Docker
- Needs PyTorch, CUDA or ROCm compiler (nvcc/hipcc), ninja
- Models: PyTorch models, Hugging Face Transformers
megatron-lm
Megatron Core is a Python library of transformer building blocks with tensor, pipeline, data, expert and context parallelism, plus FP16, BF16, FP8 and FP4 mixed precision. The repo also ships Megatron-LM, a reference set of pre-configured training scripts built on it. The README reports training models from 2B to 462B parameters, reaching up to 47% MFU on H100 clusters.
Who it is for: ML engineers and research teams training large LLMs on GPU clusters
Strengths
- Supports TP, PP, DP, EP and CP parallelism for multi-node training
- Mixed precision down to FP8 and FP4
- Benchmarked at 462B parameters on 6,144 H100 GPUs, up to 47% MFU
- Includes MoE, hybrid Mamba, post-training and RL code
Weaknesses
- Aimed at multi-GPU NVIDIA clusters; not practical for single-machine hobby fine-tuning
- Source builds can run out of memory; MAX_JOBS must be limited
- Python 3.10 support dropped; downstream projects need Python 3.12 or later
- Some features, such as DeepSeek-V4, are only on the experimental dev branch
- GPU required
- Docker
- Needs PyTorch, NVIDIA GPUs, NGC container (optional)
- Models: GPT, Mixture-of-Experts, DeepSeek-V3, Qwen3, Mixtral
More in Training and fine-tuning
Building with these?
We help teams pick an open-source AI stack and run it in production. Client work never changes a rank.