deepspeed vs peft
Two of the top projects in Training and fine-tuning, side by side: who leads on each score part and who each one suits, then setup, license, activity and what each review found.
deepspeed
PyTorch library for distributed training of large models with ZeRO
peft
Python library for LoRA and other parameter-efficient fine-tuning of Transformers models
Verdict
deepspeed ranks higher, 82 to 78. Of the 4 score parts we measure for both, deepspeed leads on 1, peft on 2 and 1 is even.
- Adoptiondeepspeed leads by 29 points, 78 to 49: a wide gap.
- FreshnessEven at 100.
- Maintenancepeft leads by 7 points, 88 to 81.
- Agent-readypeft leads by 25 points, 70 to 45: a wide gap.
Pick deepspeed if you are
ML engineers training or fine-tuning large models on multi-GPU clusters
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers/Accelerate, Lightning, Composer, Determined and MMEngine
Pick peft if you are
ML engineers fine-tuning LLMs and diffusion models on limited GPU memory
- Adapter checkpoints are MBs instead of GBs; 19MB vs 11GB for T0_3B
- LoRA trained a 12B model (mt0-xxl) on one 80GB GPU where full finetuning ran out of memory
| What we compare | deepspeed | peft |
|---|---|---|
| Score parts, out of 100 | ||
| Adoption | 78, popular | 49, known |
| Freshness | 100, active | 100, active |
| Maintenance | 81, healthy | 88, healthy |
| Easy to run | Not scored for this kind | Not scored for this kind |
| Agent-ready | 45, minimal | 70, partly |
| Facts from GitHub and the README | ||
| Stars | 43.2k | 21.8k |
| License | Apache-2.0 (permissive) | Apache-2.0 (permissive) |
| Last commit | Oct 2026 | Oct 2026 |
| Last release | Sep 2026 | Oct 2026 |
| Language | Python | Python |
| Docker | Yes | Yes |
| GPU | Optional | Optional |
| arm64 or Apple Silicon | Not stated | Mentioned |
deepspeed
DeepSpeed is a PyTorch optimization library for training large deep learning models across many GPUs. It provides ZeRO memory partitioning with CPU and NVMe offload, 3D parallelism, Ulysses sequence parallelism and MoE support, configured through a JSON file. It integrates with Hugging Face Transformers and Accelerate, Lightning, MosaicML Composer, Determined and MMEngine.
Who it is for: ML engineers training or fine-tuning large models on multi-GPU clusters
Strengths
- Used to train models such as BLOOM 176B and Megatron-Turing NLG 530B
- Integrates with Hugging Face Transformers/Accelerate, Lightning, Composer, Determined and MMEngine
- Supports NVIDIA, AMD, Intel Gaudi/XPU/CPU, Huawei Ascend NPU and Tecorigin accelerators
- Installs from PyPI; C++/CUDA ops are JIT-compiled or pre-built
Weaknesses
- Needs PyTorch installed first and a CUDA or ROCm compiler for JIT ops
- Mainly tested on NVIDIA Pascal to Hopper and AMD MI100/MI200
- Windows lacks async IO and GDS support
- Large JSON config surface; ZeRO and offload tuning requires reading the docs
- GPU optional
- Docker
- Needs PyTorch, CUDA or ROCm compiler (nvcc/hipcc), ninja
- Models: PyTorch models, Hugging Face Transformers
peft
PEFT wraps a pretrained model with adapter methods such as LoRA, soft prompts and IA3 so only a small set of extra parameters is trained. Adapters are saved as small checkpoints (19MB versus 11GB for T0_3B in the README's example). It integrates with Transformers, Diffusers, Accelerate and TRL, and can be combined with quantization such as QLoRA.
Who it is for: ML engineers fine-tuning LLMs and diffusion models on limited GPU memory
Strengths
- Adapter checkpoints are MBs instead of GBs; 19MB vs 11GB for T0_3B
- LoRA trained a 12B model (mt0-xxl) on one 80GB GPU where full finetuning ran out of memory
- Integrates with Transformers, Diffusers, Accelerate and TRL; works with quantized base models
- Install with pip; get_peft_model wraps an existing model in a few lines
Weaknesses
- Library only: no training UI, scheduler or serving; training loop is your own or Trainer
- Transformers integration lacks some PEFT features, such as merging adapters into the base model
- Models without official support need manual config for target modules
- Memory figures in the README assume an A100 80GB with over 64GB CPU RAM
- GPU optional
- Docker
- Needs PyTorch, Transformers
- Models: LoRA, Soft prompts, IA3, QLoRA, Transformers models
More in Training and fine-tuning
Building with these?
We help teams pick an open-source AI stack and run it in production. Client work never changes a rank.