#1111th of 37 in Model serving
lmcache
KV cache offloading and reuse layer for LLM serving engines
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 12.1k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Python
7,688 on 31 Mar 202612,137 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
LMCache runs as a standalone daemon that stores LLM KV cache outside the inference engine, in tiers spanning CPU RAM, local disk and remote backends. Cached KV can be reused across requests, sessions and engine instances, which cuts repeated prefill and time-to-first-token on long-context, multi-turn and RAG workloads. It also supports non-prefix reuse via CacheBlend and prefill-to-decode KV transfer over NVLink, RDMA or TCP.
Who it is for: Engineers running LLM inference clusters who want to cut prefill cost
Strengths
- Runs as a separate daemon, so KV cache survives inference engine crashes
- Backends include CPU RAM, disk, Redis/Valkey, Mooncake, S3-compatible, NIXL and GDS
- Non-prefix KV reuse at any prompt position through CacheBlend
- Exposes KV-cache metrics such as request-level and token-level prefix hits
Weaknesses
- Not a standalone server; it needs a supported serving engine in front
- README gives no hardware requirements or supported-engine version list
- Multi-node and PD disaggregation setups need external transports and storage
- No Docker or compose files detected in the repository
What it needs
- Docker
- Needs Redis/Valkey (optional), Mooncake (optional), NIXL (optional), S3-compatible storage (optional)
- Models: gpt-oss 20B/120B, multimodal models in vLLM V1
Also in Model serving
See all 37| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 77 out of 100 |