#99th of 19 in Training and fine-tuning
megatron-lm
Distributed training library for large transformer models on NVIDIA GPUs
- Stars
- 18.1k
- License
- custom license
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Megatron Core is a Python library of transformer building blocks with tensor, pipeline, data, expert and context parallelism, plus FP16, BF16, FP8 and FP4 mixed precision. The repo also ships Megatron-LM, a reference set of pre-configured training scripts built on it. The README reports training models from 2B to 462B parameters, reaching up to 47% MFU on H100 clusters.
Who it is for: ML engineers and research teams training large LLMs on GPU clusters
Strengths
- Supports TP, PP, DP, EP and CP parallelism for multi-node training
- Mixed precision down to FP8 and FP4
- Benchmarked at 462B parameters on 6,144 H100 GPUs, up to 47% MFU
- Includes MoE, hybrid Mamba, post-training and RL code
Weaknesses
- Aimed at multi-GPU NVIDIA clusters; not practical for single-machine hobby fine-tuning
- Source builds can run out of memory; MAX_JOBS must be limited
- Python 3.10 support dropped; downstream projects need Python 3.12 or later
- Some features, such as DeepSeek-V4, are only on the experimental dev branch
What it needs
- GPU required
- Docker
- Needs PyTorch, NVIDIA GPUs, NGC container (optional)
- Models: GPT, Mixture-of-Experts, DeepSeek-V3, Qwen3, Mixtral
Also in Training and fine-tuning
See all 19| Rank | Project | Score |
|---|---|---|
| 1 | unslothDesktop and web app to run, fine-tune and export open models | 88 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 82 out of 100 |
| 3 | verlReinforcement learning training library for LLM post-training with PPO and GRPO | 77 out of 100 |
| #4 | llamafactoryFine-tuning toolkit for 100+ LLMs with a CLI and Gradio web UI | 75 out of 100 |
| #5 | peftPython library for LoRA and other parameter-efficient fine-tuning of Transformers models | 74 out of 100 |