#6666th of 76 in Model serving
litserve
Python framework for writing custom inference servers on FastAPI
- Stars
- 4k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
3,865 on 25 Apr 20264,010 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Lightning AI is hiring: 50 open roles
Overview
LitServe is a Python library where you subclass LitAPI, implement setup() and predict(), and serve it through a FastAPI-based LitServer. It covers batching, streaming, multi-worker and GPU device handling, OpenAI-spec compatibility and MCP server support. It suits multi-model pipelines, agents and RAG APIs. The README says it is not a drop-in vLLM or Ollama replacement.
Who it is for: Python engineers building custom model, agent or RAG inference APIs
Strengths
- Arbitrary Python in setup() and predict(), so multi-model, agent and RAG pipelines fit in one class
- Built-in batching, streaming, worker autoscaling and OpenAI-spec endpoint support
- Framework-agnostic: PyTorch, JAX, TensorFlow, or an external engine such as vLLM
- Runs locally with pip install litserve and python server.py
Weaknesses
- Not a ready-to-run LLM server; you write the serving code for each model
- Authentication is DIY when self-hosted; token and password auth are listed for managed only
- Load balancing, scale-to-zero and multi-node inference are listed as managed-only features
- The 2x-faster-than-FastAPI claim comes from the project's own benchmarks
What it needs
- GPU optional
- Needs FastAPI
- Models: PyTorch, JAX, TensorFlow, Hugging Face, vLLM
- port 8000
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in Model serving
See all 76| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaRuns open models locally through a CLI and REST API | 92 out of 100 |
| 3 | LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs | 91 out of 100 |
| #4 | vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API | 88 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 88 out of 100 |