peft vs megatron-lm
Two of the top projects in Training and fine-tuning, side by side: who leads on each score part and who each one suits, then setup, license, activity and what each review found.
peft
Python library for LoRA and other parameter-efficient fine-tuning of Transformers models
megatron-lm
Distributed training library for large transformer models on NVIDIA GPUs
Verdict
peft ranks higher, 90 to 88. Of the 4 score parts we measure for both, peft leads on 2, megatron-lm on 1 and 1 is even.
- Adoptionpeft leads by 4 points, 88 to 84: close to even.
- FreshnessEven at 100.
- Maintenancepeft leads by 9 points, 90 to 81.
- Agent-readymegatron-lm leads by 15 points, 85 to 70: a wide gap.
Pick peft if you are
ML engineers fine-tuning LLMs and diffusion models on limited GPU memory
- Adapter checkpoints are MBs instead of GBs; 19MB vs 11GB for T0_3B
- LoRA trained a 12B model (mt0-xxl) on one 80GB GPU where full finetuning ran out of memory
Pick megatron-lm if you are
ML engineers and research teams training large LLMs on GPU clusters
- Supports TP, PP, DP, EP and CP parallelism for multi-node training
- Mixed precision down to FP8 and FP4
| What we compare | peft | megatron-lm |
|---|---|---|
| Score parts, out of 100 | ||
| Adoption | 88, widely used | 84, widely used |
| Freshness | 100, active | 100, active |
| Maintenance | 90, healthy | 81, healthy |
| Agent-ready | 70, partly | 85, ready |
| Facts from GitHub and the README | ||
| Stars | 21.9k | 18.4k |
| License | Apache-2.0 (permissive) | custom license (read the license) |
| Last commit | Nov 2026 | Nov 2026 |
| Last release | Nov 2026 | Nov 2026 |
| Language | Python | Python |
| Docker | Yes | Yes |
| GPU | Optional | Required |
| arm64 or Apple Silicon | Mentioned | Not stated |
peft
PEFT wraps a pretrained model with adapter methods such as LoRA, soft prompts and IA3 so only a small set of extra parameters is trained. Adapters are saved as small checkpoints (19MB versus 11GB for T0_3B in the README's example). It integrates with Transformers, Diffusers, Accelerate and TRL, and can be combined with quantization such as QLoRA.
Who it is for: ML engineers fine-tuning LLMs and diffusion models on limited GPU memory
Strengths
- Adapter checkpoints are MBs instead of GBs; 19MB vs 11GB for T0_3B
- LoRA trained a 12B model (mt0-xxl) on one 80GB GPU where full finetuning ran out of memory
- Integrates with Transformers, Diffusers, Accelerate and TRL; works with quantized base models
- Install with pip; get_peft_model wraps an existing model in a few lines
Weaknesses
- Library only: no training UI, scheduler or serving; training loop is your own or Trainer
- Transformers integration lacks some PEFT features, such as merging adapters into the base model
- Models without official support need manual config for target modules
- Memory figures in the README assume an A100 80GB with over 64GB CPU RAM
- GPU optional
- Docker
- Needs PyTorch, Transformers
- Models: LoRA, Soft prompts, IA3, QLoRA, Transformers models
megatron-lm
Megatron Core is a Python library of transformer building blocks with tensor, pipeline, data, expert and context parallelism, plus FP16, BF16, FP8 and FP4 mixed precision. The repo also ships Megatron-LM, a reference set of pre-configured training scripts built on it. The README reports training models from 2B to 462B parameters, reaching up to 47% MFU on H100 clusters.
Who it is for: ML engineers and research teams training large LLMs on GPU clusters
Strengths
- Supports TP, PP, DP, EP and CP parallelism for multi-node training
- Mixed precision down to FP8 and FP4
- Benchmarked at 462B parameters on 6,144 H100 GPUs, up to 47% MFU
- Includes MoE, hybrid Mamba, post-training and RL code
Weaknesses
- Aimed at multi-GPU NVIDIA clusters; not practical for single-machine hobby fine-tuning
- Source builds can run out of memory; MAX_JOBS must be limited
- Python 3.10 support dropped; downstream projects need Python 3.12 or later
- Some features, such as DeepSeek-V4, are only on the experimental dev branch
- GPU required
- Docker
- Needs PyTorch, NVIDIA GPUs, NGC container (optional)
- Models: GPT, Mixture-of-Experts, DeepSeek-V3, Qwen3, Mixtral
More in Training and fine-tuning
Building with these?
We help teams pick an open-source AI stack and run it in production. Client work never changes a rank.