22nd of 5 in Training and fine-tuning
deepspeed
PyTorch library for distributed training of large models with ZeRO
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 43.2k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
41,563 on 24 Mar 202643,229 on 13 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
DeepSpeed is a PyTorch optimization library for training large deep learning models across many GPUs. It provides ZeRO memory partitioning with CPU and NVMe offload, 3D parallelism, Ulysses sequence parallelism and MoE support, configured through a JSON file. It integrates with Hugging Face Transformers and Accelerate, Lightning, MosaicML Composer, Determined and MMEngine.
Who it is for: ML engineers training or fine-tuning large models on multi-GPU clusters
Strengths
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers/Accelerate, Lightning, Composer, Determined and MMEngine
- Supports NVIDIA, AMD, Intel Gaudi/XPU/CPU, Huawei Ascend NPU and Tecorigin accelerators
- Installs from PyPI; C++/CUDA ops are JIT-compiled or pre-built
Weaknesses
- Needs PyTorch installed first and a CUDA or ROCm compiler for JIT ops
- Mainly tested on NVIDIA Pascal to Hopper and AMD MI100/MI200
- Windows lacks async IO and GDS support
- Large JSON config surface; ZeRO and offload tuning requires reading the docs
What it needs
- GPU optional
- Docker
- Needs PyTorch, CUDA or ROCm compiler (nvcc/hipcc), ninja
- Models: PyTorch models, Hugging Face Transformers
Head to head
Also in Training and fine-tuning
See all 5| Rank | Project | Score |
|---|---|---|
| 1 | llamafactoryFine-tuning toolkit for 100+ LLMs with a CLI and Gradio web UI | 74 out of 100 |
| 3 | minimindTrain a 64M-parameter LLM from scratch in plain PyTorch | 64 out of 100 |
| #4 | nanochatMinimal PyTorch codebase to train and chat with a small GPT-2-class LLM | 49 out of 100 |
| #5 | colossalaiDistributed training library for large models with many parallelism strategies | 43 out of 100 |