peft vs megatron-lm

Two of the top projects in Training and fine-tuning, side by side: who leads on each score part and who each one suits, then setup, license, activity and what each review found.

11st of 99 in Training and fine-tuning

peft

Python library for LoRA and other parameter-efficient fine-tuning of Transformers models

#44th of 99 in Training and fine-tuning

megatron-lm

Distributed training library for large transformer models on NVIDIA GPUs

Verdict

peft ranks higher, 90 to 88. Of the 4 score parts we measure for both, peft leads on 2, megatron-lm on 1 and 1 is even.

  • Adoptionpeft leads by 4 points, 88 to 84: close to even.
  • FreshnessEven at 100.
  • Maintenancepeft leads by 9 points, 90 to 81.
  • Agent-readymegatron-lm leads by 15 points, 85 to 70: a wide gap.

Pick peft if you are

ML engineers fine-tuning LLMs and diffusion models on limited GPU memory

  • Adapter checkpoints are MBs instead of GBs; 19MB vs 11GB for T0_3B
  • LoRA trained a 12B model (mt0-xxl) on one 80GB GPU where full finetuning ran out of memory

Pick megatron-lm if you are

ML engineers and research teams training large LLMs on GPU clusters

  • Supports TP, PP, DP, EP and CP parallelism for multi-node training
  • Mixed precision down to FP8 and FP4
peft vs megatron-lm: score parts and facts
What we comparepeftmegatron-lm
Score parts, out of 100
Adoption88, widely used84, widely used
Freshness100, active100, active
Maintenance90, healthy81, healthy
Agent-ready70, partly85, ready
Facts from GitHub and the README
Stars21.9k18.4k
LicenseApache-2.0 (permissive)custom license (read the license)
Last commitNov 2026Nov 2026
Last releaseNov 2026Nov 2026
LanguagePythonPython
DockerYesYes
GPUOptionalRequired
arm64 or Apple SiliconMentionedNot stated

peft

PEFT wraps a pretrained model with adapter methods such as LoRA, soft prompts and IA3 so only a small set of extra parameters is trained. Adapters are saved as small checkpoints (19MB versus 11GB for T0_3B in the README's example). It integrates with Transformers, Diffusers, Accelerate and TRL, and can be combined with quantization such as QLoRA.

Who it is for: ML engineers fine-tuning LLMs and diffusion models on limited GPU memory

Strengths

  • Adapter checkpoints are MBs instead of GBs; 19MB vs 11GB for T0_3B
  • LoRA trained a 12B model (mt0-xxl) on one 80GB GPU where full finetuning ran out of memory
  • Integrates with Transformers, Diffusers, Accelerate and TRL; works with quantized base models
  • Install with pip; get_peft_model wraps an existing model in a few lines

Weaknesses

  • Library only: no training UI, scheduler or serving; training loop is your own or Trainer
  • Transformers integration lacks some PEFT features, such as merging adapters into the base model
  • Models without official support need manual config for target modules
  • Memory figures in the README assume an A100 80GB with over 64GB CPU RAM
  • GPU optional
  • Docker
  • Needs PyTorch, Transformers
  • Models: LoRA, Soft prompts, IA3, QLoRA, Transformers models

megatron-lm

Megatron Core is a Python library of transformer building blocks with tensor, pipeline, data, expert and context parallelism, plus FP16, BF16, FP8 and FP4 mixed precision. The repo also ships Megatron-LM, a reference set of pre-configured training scripts built on it. The README reports training models from 2B to 462B parameters, reaching up to 47% MFU on H100 clusters.

Who it is for: ML engineers and research teams training large LLMs on GPU clusters

Strengths

  • Supports TP, PP, DP, EP and CP parallelism for multi-node training
  • Mixed precision down to FP8 and FP4
  • Benchmarked at 462B parameters on 6,144 H100 GPUs, up to 47% MFU
  • Includes MoE, hybrid Mamba, post-training and RL code

Weaknesses

  • Aimed at multi-GPU NVIDIA clusters; not practical for single-machine hobby fine-tuning
  • Source builds can run out of memory; MAX_JOBS must be limited
  • Python 3.10 support dropped; downstream projects need Python 3.12 or later
  • Some features, such as DeepSeek-V4, are only on the experimental dev branch
  • GPU required
  • Docker
  • Needs PyTorch, NVIDIA GPUs, NGC container (optional)
  • Models: GPT, Mixture-of-Experts, DeepSeek-V3, Qwen3, Mixtral

More in Training and fine-tuning

Building with these?

We help teams pick an open-source AI stack and run it in production. Client work never changes a rank.