#6464th of 73 in Model serving
lightllm
Python LLM inference server with token-level KV cache management
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.4k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Aug 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
LightLLM is a pure-Python inference and serving framework for large language models. It builds on ideas and kernels from FasterTransformer, TGI, vLLM, SGLang, FlashAttention and OpenAI Triton. The README highlights DeepSeek-R1 serving on a single H200 machine, constrained decoding for structured output, a Past-Future request scheduler, and prefix KV cache transfer between data-parallel ranks.
Who it is for: Engineers serving LLMs on GPUs, and researchers building on a Python engine
Strengths
- Pure-Python code with token-level KV cache management, easy to modify for research
- Some kernels are reused by vLLM, SGLang and Aphrodite, per the README
- Published scheduler (ASPLOS'25) and constrained decoding (ACL2025) research
- Docs include a DeepSeek deployment tutorial
Weaknesses
- README lists no supported model families or quantization formats
- No Dockerfile or compose file detected in the repository
- README gives no benchmark numbers, only links to blog posts
- README shows no API endpoints or launch commands
What it needs
- Needs Triton, FlashAttention, flashinfer
- Models: DeepSeek-R1, Qwen3-8B
Also in Model serving
See all 73| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 91 out of 100 |
| #4 | vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API | 88 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 88 out of 100 |