#8181st of 105 in Model serving

production-stack

Helm-deployed vLLM cluster with request router and Grafana monitoring

Releases and problems feed (RSS)

Stars
2.7k
License
Apache-2.0
Last commit
Nov 2026
Last release
Nov 2026
Language
Python
Stars, last 29 weeks+440
29 Apr 2026: 2,289 stars6 May 2026: 2,302 stars13 May 2026: 2,318 stars20 May 2026: 2,328 stars27 May 2026: 2,342 stars3 Jun 2026: 2,360 stars10 Jun 2026: 2,375 stars17 Jun 2026: 2,393 stars24 Jun 2026: 2,406 stars1 Jul 2026: 2,418 stars8 Jul 2026: 2,429 stars15 Jul 2026: 2,440 stars22 Jul 2026: 2,453 stars29 Jul 2026: 2,467 stars5 Aug 2026: 2,482 stars12 Aug 2026: 2,495 stars19 Aug 2026: 2,510 stars26 Aug 2026: 2,521 stars2 Sept 2026: 2,546 stars9 Sept 2026: 2,559 stars16 Sept 2026: 2,571 stars23 Sept 2026: 2,625 stars30 Sept 2026: 2,643 stars7 Oct 2026: 2,654 stars14 Oct 2026: 2,665 stars21 Oct 2026: 2,680 stars28 Oct 2026: 2,695 stars4 Nov 2026: 2,707 stars11 Nov 2026: 2,723 stars18 Nov 2026: 2,729 stars

2,289 on 29 Apr 20262,729 on 18 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Inferact is hiring: 32 open roles

Overview

Reference implementation for running vLLM on Kubernetes, packaged as a Helm chart. It combines vLLM serving engines, a request router that uses session IDs or round-robin to improve KV cache reuse, and a Prometheus and Grafana observability stack. The deployed stack exposes the same OpenAI-compatible API as vLLM, with optional KV cache offloading through LMCache.

Who it is for: Platform engineers running multi-GPU vLLM inference on Kubernetes

Strengths

  • Scales from one vLLM instance to many without application code changes
  • Router supports round-robin and session-ID routing, with Kubernetes service discovery
  • Grafana dashboard shows TTFT, latency, running and pending requests, and KV cache hit rate
  • Tutorials cover AWS, GCP, Azure and Lambda Labs, plus multi-model and LMCache KV offloading

Weaknesses

  • Requires a Kubernetes cluster with GPUs; no non-Kubernetes install path
  • Prefix-aware routing is marked work in progress
  • Autoscaling and disaggregated prefill are still roadmap items
  • Router is written in Python; a faster non-Python router is only planned

What it needs

  • GPU required
  • Docker
  • Needs Kubernetes, Helm, Prometheus, Grafana, LMCache (optional)
  • Models: vLLM-supported models

What changed

From the release notes. No flag means the note does not say so, not that the upgrade is safe.

Known problems

Open now, reported in the last 90 days

Reported, then fixed by a linked change

Titles are the reporters' own words, taken from the project's GitHub issues and ranked by reactions. We have not tried to reproduce any of them.

Also in Model serving

See all 105
Also in Model serving
RankProjectScore
1llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server135.9k starsMIT94 out of 100
2OllamaRuns open models locally through a CLI and REST API187.8k starsMITknown advisory in latest version91 out of 100
3LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs50.5k starsMIT91 out of 100
#4strataDesktop and terminal app that runs GGUF models locally with OpenAI-compatible APIs46.4k stars+16,488 (+55%) in 30 daysMIT89 out of 100
#5modularMojo language and MAX inference framework with an OpenAI-compatible server30.2k stars+228 (+0.8%) in 30 dayscustom license89 out of 100