#5252nd of 71 in Model serving
llm-d
Kubernetes stack for distributed LLM inference on top of vLLM and SGLang
- Stars
- 5.2k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- Python
3,023 on 22 Apr 20265,237 on 11 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
llm-d sits above model servers such as vLLM and SGLang and adds prefix-cache and load-aware routing, tiered KV-cache offload to CPU or disk, and prefill/decode disaggregation with wide expert-parallelism. It also covers multi-tenant flow control, session-aware routing for agents, OpenAI-compatible batch APIs and SLO-aware autoscaling. It deploys through benchmarked Helm-chart guides and is a CNCF sandbox project.
Who it is for: Platform teams running multi-GPU LLM inference on Kubernetes in production
Strengths
- Prefix-cache-aware routing reported at 3x output throughput versus round-robin on Llama 3.1 70B
- Supports NVIDIA, AMD and Intel GPUs and Google TPUs, with vLLM and SGLang backends
- Prefill/decode disaggregation and tiered KV offload for large models like gpt-oss-120b
- Tested Helm-chart guides with reproducible benchmarks published on Prism
Weaknesses
- Requires a Kubernetes cluster; not a single-node or laptop server
- Needs a separate model server (vLLM or SGLang) underneath
- Predicted-latency scheduling is described as experimental in the README
- Many components (router, flow control, KV indexing) add operational overhead for small deployments
What it needs
- GPU required
- Docker
- Needs Kubernetes, vLLM or SGLang, Helm
- Models: DeepSeek-V4, Kimi K3, gpt-oss-120b, Llama 3.3 70B, Llama 3.1 70B
What changed
- v1.0.0 ↗ (opens in a new tab)Security fix
“Revise security vulnerability reporting methods by @davidgs in https://github.com/llm-d/llm-d/pull/2692”
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in Model serving
See all 71| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 91 out of 100 |
| #4 | vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API | 88 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 88 out of 100 |