#4949th of 57 in Model serving
cactus
On-device LLM, vision and speech inference engine for phones and wearables
- Stars
- 6.2k
- License
- custom license
- Last commit
- Sep 2026
- Last release
- Sep 2026
- Language
- C++
4,507 on 3 Apr 20266,171 on 30 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Cactus is a C++ inference engine with its own computation graph, ARM/Apple/Android kernels and rotation-based quantization (CQ, 1 to 4 bits). It runs text, vision and speech models on phones, Macs and wearables, and can hand low-confidence queries off to a cloud model. It ships a CLI with an OpenAI-compatible local server (default port 8080) and bindings for Swift, Kotlin, Flutter, React Native, Python and Rust.
Who it is for: Mobile and edge developers embedding local LLM, vision and speech models in apps
Strengths
- Published benchmarks per device, e.g. Gemma-4-E2B-CQ4 decodes 37 tok/s on iPhone 17 Pro
- Bindings for Swift, Kotlin, Flutter, React Native, Python and Rust for mobile apps
- Quantization accuracy tables across bit widths (CQ4 close to F16 on most listed tasks)
- Confidence-based cloud handoff for hard queries, configurable threshold and timeout
Weaknesses
- License is reported as NOASSERTION; terms need checking before commercial use
- Converting arbitrary HuggingFace models is labelled experimental
- Only Liquid, Gemma, Whisper, Parakeet and Qwen families are described as well tested
- Benchmarks cover Apple devices only; no Android or Linux numbers in the README
What it needs
- GPU optional
- Needs HuggingFace (model download/convert), cmake and build-essential (Linux build), Python 3.12 (setup/CLI)
- Models: Gemma, Liquid LFM, Qwen, Whisper, Parakeet
- port 8080
Quick answers
Is cactus open source?
Its code is public on GitHub under its own license terms; read them before you use it at work.
Does cactus need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Model serving
See all 57| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 91 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |