#4545th of 47 in Model serving
powerinfer
CPU/GPU hybrid LLM inference engine exploiting sparse neuron activation
- Stars
- 9.8k
- License
- MIT
- Last commit
- May 2026
- Language
- C++
9,151 on 1 Apr 20269,844 on 21 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
PowerInfer is a C++ inference engine, derived from llama.cpp, that keeps frequently activated "hot" neurons on the GPU and computes the rest on the CPU. It runs ReLU-sparse models in a GGUF variant (*.powerinfer.gguf) that bundles predictor weights, and uses --vram-budget in place of -ngl. The README reports up to 11x speedup over llama.cpp on Falcon(ReLU)-40B on one RTX 4090.
Who it is for: Engineers running large ReLU-sparse LLMs on a single consumer GPU plus CPU
Strengths
- Hot/cold neuron split lets 40B-70B models run on one consumer GPU with CPU offload
- Reports up to 11x over llama.cpp on Falcon-40B (RTX 4090), up to 8x with INT4 on RTX 2080Ti
- Keeps llama.cpp-style examples such as server, batched generation and perplexity
- NVIDIA (CUDA), AMD (ROCm) and CPU-only builds on Linux and Windows
Weaknesses
- Only ReLU/ReGLU/Squared ReLU models work; Mistral, original Llama and Qwen are unsupported
- Needs PowerInfer GGUF files with predictor weights; llama.cpp weights give no speedup
- Short model list: Falcon-40B, Llama 2 family, ProSparse Llama 2, Bamboo-7B
- macOS is CPU only with little gain; no tagged release
What it needs
- GPU optional
- Docker
- Needs CMake 3.17+, Python 3.8+, pip 19.3+, CUDA or ROCm (for GPU builds)
- Models: PowerInfer GGUF, ReluLLaMA 7B/13B/70B, ReluFalcon-40B, ProSparse-LLaMA-2, Bamboo-7B
Also in Model serving
See all 47| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 79 out of 100 |
| #5 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |