#3030th of 33 in Model serving
nano-vllm
Minimal vLLM-style inference engine in about 1,200 lines of Python
- Stars
- 15.7k
- License
- MIT
- Last commit
- Apr 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Nano-vLLM is a from-scratch reimplementation of vLLM for offline batch inference, with an LLM.generate API that mirrors vLLM's. It includes prefix caching, tensor parallelism, Torch compilation and CUDA graphs. The README benchmarks Qwen3-0.6B on an RTX 4070 Laptop (8GB) at 1434 tokens/s against vLLM's 1362 tokens/s.
Who it is for: Engineers who want to read or hack on a compact LLM inference engine
Strengths
- Small codebase (~1,200 lines of Python) that is practical to read and modify
- API mirrors vLLM's LLM and SamplingParams interface
- Includes prefix caching, tensor parallelism, Torch compile and CUDA graphs
- README benchmark shows throughput on par with vLLM on one small-model setup
Weaknesses
- Offline inference only; README describes no HTTP or OpenAI-compatible server
- Benchmark covers one model and one laptop GPU; broader results are unknown
- README lists only Qwen3-0.6B; other supported architectures are unknown
- No tagged releases; installed directly from the GitHub repository
What it needs
- Needs Hugging Face model weights
- Models: Qwen3-0.6B
Also in Model serving
See all 33| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 73 out of 100 |