#2727th of 47 in Model serving
dynamo
Multi-node orchestration layer for vLLM, SGLang and TensorRT-LLM inference
- Stars
- 8.4k
- License
- custom license
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Rust
6,397 on 1 Apr 20268,354 on 21 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Dynamo sits above existing inference engines (vLLM, SGLang, TensorRT-LLM) and coordinates them across GPUs and nodes. It provides disaggregated prefill/decode, KV-cache-aware routing, tiered KV offload (GPU, CPU, SSD, remote storage) and an SLA-driven planner for autoscaling, behind an OpenAI-compatible frontend. The core is written in Rust with Python bindings; Kubernetes is the recommended production target.
Who it is for: Teams serving LLMs across many GPUs or nodes needing routing and autoscaling
Strengths
- Adds disaggregated prefill/decode and KV-aware routing on top of vLLM, SGLang and TensorRT-LLM
- KV block manager offloads cache across GPU, CPU, SSD and remote storage such as S3 or Azure blob
- SLA-based planner autoscaler and Kubernetes operator with gang scheduling (Grove)
- No etcd or NATS needed for local or Kubernetes deployments
Weaknesses
- Not useful for a single model on a single GPU; the README says the engine alone suffices
- Production path assumes Kubernetes and multi-GPU clusters; heavy for small setups
- Zero-config deploy (DGDR) is marked beta
- KVBM support for SGLang is still marked in progress
What it needs
- GPU required
- Docker
- Needs vLLM, SGLang or TensorRT-LLM, Kubernetes (production)
- Models: vLLM, SGLang, TensorRT-LLM, OpenAI-compatible API, multimodal
- port 8000
Also in Model serving
See all 47| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 79 out of 100 |
| #5 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |