#2020th of 23 in Open models
rwkv-lm
Reference code for training and running RWKV, an attention-free RNN language model
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 14.7k
- License
- Apache-2.0
- Last commit
- Sep 2026
- Last release
- Dec 2023
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Reference implementation of RWKV-7 (and earlier v5/v6), an RNN-style language model that trains in parallel like a GPT and decodes with constant-size state and no KV cache. The repo has PyTorch and CUDA training scripts for MiniPile and Pile, demo inference in GPT and RNN modes, and fine-tuning notes such as state-tuning. Weights, GUIs and servers live in linked separate projects.
Who it is for: Researchers and engineers training or studying attention-free RNN LLMs
Strengths
- Constant-memory decoding with no KV cache; state is about 1.6M params at 24 layers, 1024 dim
- Reference training code with CUDA kernels; default config needs one GPU with about 7 GB VRAM
- Same model runs in parallel GPT mode or sequential RNN mode
- Links to weights, GGUF builds, mobile runtime, Runner GUI and PEFT/RLHF forks
Weaknesses
- No packaged RWKV-7 layer; init, weight decay and per-parameter LR must be copied from train_temp
- README says the FLA RWKV-7 implementation is not aligned with the reference and performs worse
- Research-style repo: no Docker, no API server, last tagged release is from 2023-12
- README is a long list of links; RWKV-5/6 and RWKV-7 instructions are mixed together
What it needs
- GPU required
- Needs PyTorch 2.5+, CUDA 12.4+, DeepSpeed, pytorch-lightning 1.9.5, wandb, ninja
- Models: RWKV-7, RWKV-6, RWKV-5, PyTorch .pth, GGUF (linked)
Also in Open models
See all 23| Rank | Project | Score |
|---|---|---|
| 1 | llmfitRust tool that checks which LLMs fit your RAM, CPU and GPU | 86 out of 100 |
| 2 | layaSingle-pass multilingual classifier for typed choice, score and yes/no decisions | 83 out of 100 |
| 3 | timesfmPretrained time-series foundation model for zero-shot forecasting from Google Research | 74 out of 100 |
| #4 | unlimited-ocrOCR model that parses long documents and multi-page PDFs in one pass | 69 out of 100 |
| #5 | speechPyTorch toolkit for training and deploying ASR, TTS and speech LLM models | 67 out of 100 |