#1616th of 22 in Training and fine-tuning
openrlhf
Ray and vLLM based RLHF training framework for LLMs and VLMs
- Stars
- 10.1k
- License
- Apache-2.0
- Last commit
- Sep 2026
- Last release
- Sep 2026
- Language
- Python
9,200 on 31 Mar 202610,104 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
OpenRLHF trains language and vision-language models with reinforcement learning from human feedback. It uses Ray to schedule actor, reward, reference and critic models across GPUs, vLLM for sample generation, and DeepSpeed ZeRO-3 for training. It implements PPO, REINFORCE++, GRPO and RLOO, plus SFT, DPO and reward model training, with single-turn or multi-turn agent rollouts.
Who it is for: ML engineers running RL fine-tuning of LLMs on multi-GPU clusters
Strengths
- Several RL algorithms (PPO, REINFORCE++, GRPO, RLOO) selectable via one estimator flag
- Hybrid engine lets models and vLLM share GPUs; README cites 70B+ scale
- Multi-turn agent mode with reset/step environments, async training and VLM support
- Also covers SFT, DPO and reward models, with LoRA/QLoRA and sample packing
Weaknesses
- Needs NVIDIA GPUs and a Ray, vLLM and DeepSpeed stack; setup is involved
- README recommends a specific NVIDIA PyTorch container and removing conflicting pip packages
- Large CLI flag surface; configuration lives mostly in example shell scripts
- Hardware requirements per model size are not stated in the README
What it needs
- GPU required
- Needs Ray, vLLM, DeepSpeed, HuggingFace Transformers, NCCL
- Models: HuggingFace Transformers models, VLMs (e.g. Qwen3.5), MoE, LoRA/QLoRA
Also in Training and fine-tuning
See all 22| Rank | Project | Score |
|---|---|---|
| 1 | unslothDesktop and web app to run, fine-tune and export open models | 88 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 83 out of 100 |
| 3 | verlReinforcement learning training library for LLM post-training with PPO and GRPO | 82 out of 100 |
| #4 | peftPython library for LoRA and other parameter-efficient fine-tuning of Transformers models | 81 out of 100 |
| #5 | trlPost-training library with SFT, DPO, GRPO and KTO trainers | 78 out of 100 |