#4747th of 47 in Model serving
gemma.cpp
Minimal C++ CPU inference engine for Gemma 2, Gemma 3 and PaliGemma 2
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 7.1k
- License
- Apache-2.0
- Last commit
- Oct 2025
- Last release
- Mar 2025
- Language
- C++
Missed the latest GitHub check; these facts may be older.
Overview
gemma.cpp runs Gemma 2, Gemma 3 and PaliGemma 2 models on CPU using Google Highway portable SIMD. It has a ~2K LoC core and ships an interactive terminal app, C++ APIs with streaming and batching, and basic Python bindings. It targets research and experimentation rather than production deployment, and the README points to JAX, Keras, PyTorch or Transformers for that.
Who it is for: Researchers and C++ developers experimenting with Gemma inference on CPU
Strengths
- Small codebase (~2K LoC core, ~4K LoC utilities), easy to read and modify
- Compressed weights: 8-bit switched floating point, bf16, f32 and non-uniform 4-bit
- Embeds as a library via CMake FetchContent with few dependencies
- Includes backward pass (VJP) and Adam optimizer for Gemma research
Weaknesses
- CPU-only inference; no GPU support
- Supports only Gemma 2, Gemma 3 and PaliGemma 2 models
- No HTTP server; frontends are a terminal app, C++ API and basic Python bindings
- Weights need Kaggle download with consent form; Safetensors converter not open sourced
What it needs
- no GPU
- Needs CMake, Clang (C++17), Google Highway, SentencePiece
- Models: Gemma 2, Gemma 3, PaliGemma 2, .sbs compressed weights
Also in Model serving
See all 47| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 79 out of 100 |
| #5 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |