#2828th of 57 in Model serving
mooncake
KV cache transfer engine and distributed cache store for LLM inference clusters
- Stars
- 6.9k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Aug 2026
- Language
- C++
4,985 on 4 Apr 20266,945 on 31 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Mooncake is the serving platform behind Kimi, built around a KV cache-centric split of prefill and decode clusters. It provides a Transfer Engine for batched data movement over RDMA, TCP, NVLink, EFA and other transports, plus Mooncake Store, a distributed KV cache and weight store spanning DRAM and SSD. It is integrated into SGLang, vLLM, TensorRT-LLM, LMDeploy and others.
Who it is for: Infra engineers running multi-node, prefill/decode-disaggregated LLM serving
Strengths
- Transfer Engine reports up to 190 GB/s on 8x400 Gbps RoCE, about 4.6x faster than TCP
- Integrated upstream in SGLang, vLLM, TensorRT-LLM, LMDeploy, NIXL and vLLM-Ascend
- Mooncake Store pools KV cache across DRAM and SSD/NVMe with replication and eviction
- Separate PyPI wheels for CUDA, ROCm, NPU, MUSA, EFA and non-CUDA builds
Weaknesses
- Infrastructure component, not a standalone server; needs a serving engine such as vLLM or SGLang
- Reported bandwidth figures assume RDMA NICs and multi-node GPU clusters
- Several wheel variants by CUDA version make installation choice non-trivial
What it needs
- GPU optional
- Docker
- Needs RDMA NICs (for high-bandwidth transport), SGLang or vLLM (integration)
- Models: LLMs via SGLang, vLLM, TensorRT-LLM, LMDeploy integrations
Quick answers
Is mooncake open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Can you run mooncake with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does mooncake need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Model serving
See all 57| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 91 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |