#3030th of 31 in Training and fine-tuning
gpt-neox
EleutherAI's Megatron/DeepSpeed library for pretraining large language models
- Stars
- 7.5k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Mar 2023
- Language
- Python
7,360 on 1 Apr 20267,473 on 21 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
GPT-NeoX is a training library for autoregressive language models with billions of parameters, built on NVIDIA Megatron and DeepSpeed with ZeRO and 3D parallelism. It launches through Slurm, MPI, pdsh or the IBM Job Step Manager and runs on NVIDIA and AMD GPUs. The README says it targets from-scratch large-scale training and recommends Hugging Face transformers for generic inference.
Who it is for: Teams pretraining billion-parameter LLMs on multi-GPU clusters
Strengths
- Used by 40+ organizations to train 50+ models, including Pythia and StableLM
- ZeRO and 3D parallelism, flash attention, Transformer Engine, rotary and ALiBi embeddings
- Preset configs for Pythia, PaLM, Falcon and LLaMA 1 and 2
- Exports to Hugging Face; logs to WandB, Comet or TensorBoard
Weaknesses
- Not meant for small-scale training or generic inference, per the README
- Depends on DeeperSpeed, a DeepSpeed fork; environment isolation strongly advised
- Tested on Python 3.8-3.10 and PyTorch 1.8-2.0; other versions not guaranteed
- Last tagged release is from March 2023
What it needs
- GPU required
- Docker + Compose
- Needs PyTorch, DeeperSpeed, Slurm, MPI or pdsh for multi-node, Flash Attention (optional), Transformer Engine (optional), WandB, Comet or TensorBoard (optional)
- Models: GPT-NeoX, Pythia, LLaMA 1 and 2, Falcon, PaLM
Also in Training and fine-tuning
See all 31| Rank | Project | Score |
|---|---|---|
| 1 | unslothDesktop and web app to run, fine-tune and export open models | 88 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 84 out of 100 |
| 3 | verlReinforcement learning training library for LLM post-training with PPO and GRPO | 84 out of 100 |
| #4 | peftPython library for LoRA and other parameter-efficient fine-tuning of Transformers models | 84 out of 100 |
| #5 | trlPost-training library with SFT, DPO, GRPO and KTO trainers | 81 out of 100 |