#8181st of 105 in Model serving
production-stack
Helm-deployed vLLM cluster with request router and Grafana monitoring
- Stars
- 2.7k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- Python
2,289 on 29 Apr 20262,729 on 18 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Inferact is hiring: 32 open roles
Overview
Reference implementation for running vLLM on Kubernetes, packaged as a Helm chart. It combines vLLM serving engines, a request router that uses session IDs or round-robin to improve KV cache reuse, and a Prometheus and Grafana observability stack. The deployed stack exposes the same OpenAI-compatible API as vLLM, with optional KV cache offloading through LMCache.
Who it is for: Platform engineers running multi-GPU vLLM inference on Kubernetes
Strengths
- Scales from one vLLM instance to many without application code changes
- Router supports round-robin and session-ID routing, with Kubernetes service discovery
- Grafana dashboard shows TTFT, latency, running and pending requests, and KV cache hit rate
- Tutorials cover AWS, GCP, Azure and Lambda Labs, plus multi-model and LMCache KV offloading
Weaknesses
- Requires a Kubernetes cluster with GPUs; no non-Kubernetes install path
- Prefix-aware routing is marked work in progress
- Autoscaling and disaggregated prefill are still roadmap items
- Router is written in Python; a faster non-Python router is only planned
What it needs
- GPU required
- Docker
- Needs Kubernetes, Helm, Prometheus, Grafana, LMCache (optional)
- Models: vLLM-supported models
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Known problems
Open now, reported in the last 90 days
Reported, then fixed by a linked change
- bug: Istio compression messes up the router-engine connection and user receives decompressed body with content-encoding: "gzip" ↗ (opens in a new tab)2 reactions · closed 21 Feb 2026Fixed by fix: filter hop-by-hop headers from streaming responses ↗ (opens in a new tab)
- bug: CRD VllmRuntime : increase shm size ↗ (opens in a new tab)1 reactions · closed 28 Jul 2026Fixed by feat(operator): support shmSize on VLLMRuntime for tensor parallelism (#899) ↗ (opens in a new tab)
- bug: Router - Stale Prometheus endpoint health metrics when engines become unavailable in service discovery ↗ (opens in a new tab)1 reactions · closed 14 May 2026Fixed by [Bugfix][Router] Fix stale Prometheus endpoint health metrics (#888) ↗ (opens in a new tab)
- bug: Router does not authenticate with vllmApiKey to /v1/audio/transcriptions in v0.1.10 ↗ (opens in a new tab)0 reactions · closed 23 Apr 2026Fixed by [Bugfix][Router] Fix router auth for transcription proxy ↗ (opens in a new tab)
- bug: Router log level is unchangeable, and default in log.py is DEBUG ↗ (opens in a new tab)0 reactions · closed 13 Feb 2026Fixed by fix: make --log-level CLI argument actually control router log levels ↗ (opens in a new tab)
Titles are the reporters' own words, taken from the project's GitHub issues and ranked by reactions. We have not tried to reproduce any of them.
Also in Model serving
See all 105| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaRuns open models locally through a CLI and REST API | 91 out of 100 |
| 3 | LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs | 91 out of 100 |
| #4 | strataDesktop and terminal app that runs GGUF models locally with OpenAI-compatible APIs | 89 out of 100 |
| #5 | modularMojo language and MAX inference framework with an OpenAI-compatible server | 89 out of 100 |