#55th of 5 in Training and fine-tuning
colossalai
Distributed training library for large models with many parallelism strategies
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 41.4k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Jun 2025
- Language
- Python
41,066 on 24 Mar 202641,431 on 13 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Colossal-AI is a PyTorch-based library for training and running large models across multiple GPUs. It provides data, pipeline, 1D/2D/2.5D/3D tensor and sequence parallelism, ZeRO, auto-parallelism and heterogeneous memory management. The repo also hosts example applications such as Colossal-LLaMA-2, ColossalChat (RLHF) and Stable Diffusion acceleration, plus an inference module.
Who it is for: ML engineers training or fine-tuning large models on multi-GPU clusters
Strengths
- Covers data, pipeline, tensor (1D to 3D), sequence parallelism and ZeRO in one library
- Published benchmarks for 7B and 70B Llama-like models on H200 and B200 GPUs
- Includes worked applications: Colossal-LLaMA-2, ColossalChat RLHF, Stable Diffusion training
- Apache-2.0 license, with a Dockerfile and PyPI install
Weaknesses
- Last tagged release is 2025-06-04, older than recent commits
- README is heavily promotional for the vendor's GPU cloud and model APIs
- Multi-GPU focus; RAM and minimum GPU requirements are not stated in the README
- Many example sections are dated, with news items ending in 2025
What it needs
- GPU required
- Docker
- Needs PyTorch
- Models: Llama, Open-Sora, Stable Diffusion, GPT-2, GPT-3
Also in Training and fine-tuning
See all 5| Rank | Project | Score |
|---|---|---|
| 1 | llamafactoryFine-tuning toolkit for 100+ LLMs with a CLI and Gradio web UI | 74 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 72 out of 100 |
| 3 | minimindTrain a 64M-parameter LLM from scratch in plain PyTorch | 64 out of 100 |
| #4 | nanochatMinimal PyTorch codebase to train and chat with a small GPT-2-class LLM | 49 out of 100 |