#2525th of 37 in Model serving
llama-cpp-python
Python bindings and OpenAI-compatible server for llama.cpp
- Stars
- 10.7k
- License
- MIT
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Python
10,006 on 31 Mar 202610,650 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Python package that wraps llama.cpp through ctypes, with a high-level Llama class for completion and chat on GGUF models. It also ships an OpenAI-compatible web server supporting function calling, vision models and multiple models. Installation builds llama.cpp from source via pip, with CMAKE_ARGS selecting backends such as CUDA, Metal, ROCm, Vulkan, SYCL, BLAS and RPC; pre-built wheels exist for some.
Who it is for: Python developers running GGUF models locally behind an OpenAI-style API
Strengths
- Low-level ctypes access to the C API plus a high-level Llama class
- OpenAI-compatible server with function calling, vision and multi-model config
- JSON and JSON Schema constrained output via response_format
- Pre-built wheels for CPU, CUDA, Metal, ROCm and Vulkan avoid local compilation
Weaknesses
- Default pip install compiles llama.cpp and needs a C compiler
- Only GGUF models are supported
- Pre-built GPU wheels cover limited Python versions (3.10 to 3.12) and CUDA versions
- Function calling needs a specific chat format such as chatml-function-calling or functionary
What it needs
- GPU optional
- Docker
- Needs Python 3.8+, C compiler, CMake
- Models: GGUF, llava, moondream2, minicpm-v-2.6, qwen2.5-vl
Also in Model serving
See all 37| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 77 out of 100 |