#6666th of 76 in Model serving

litserve

Python framework for writing custom inference servers on FastAPI

Stars
4k
License
Apache-2.0
Last commit
Oct 2026
Last release
Sep 2026
Language
Python
Stars, last 29 weeks+145
25 Apr 2026: 3,865 stars2 May 2026: 3,869 stars9 May 2026: 3,872 stars16 May 2026: 3,876 stars23 May 2026: 3,877 stars30 May 2026: 3,880 stars6 Jun 2026: 3,884 stars13 Jun 2026: 3,888 stars20 Jun 2026: 3,894 stars27 Jun 2026: 3,899 stars4 Jul 2026: 3,905 stars11 Jul 2026: 3,915 stars18 Jul 2026: 3,919 stars25 Jul 2026: 3,920 stars1 Aug 2026: 3,923 stars8 Aug 2026: 3,924 stars15 Aug 2026: 3,931 stars22 Aug 2026: 3,932 stars29 Aug 2026: 3,936 stars5 Sept 2026: 3,940 stars12 Sept 2026: 3,942 stars19 Sept 2026: 3,945 stars26 Sept 2026: 3,948 stars3 Oct 2026: 3,950 stars10 Oct 2026: 3,952 stars17 Oct 2026: 3,959 stars24 Oct 2026: 3,962 stars31 Oct 2026: 3,976 stars7 Nov 2026: 3,998 stars14 Nov 2026: 4,010 stars

3,865 on 25 Apr 20264,010 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Lightning AI is hiring: 50 open roles

Overview

LitServe is a Python library where you subclass LitAPI, implement setup() and predict(), and serve it through a FastAPI-based LitServer. It covers batching, streaming, multi-worker and GPU device handling, OpenAI-spec compatibility and MCP server support. It suits multi-model pipelines, agents and RAG APIs. The README says it is not a drop-in vLLM or Ollama replacement.

Who it is for: Python engineers building custom model, agent or RAG inference APIs

Strengths

  • Arbitrary Python in setup() and predict(), so multi-model, agent and RAG pipelines fit in one class
  • Built-in batching, streaming, worker autoscaling and OpenAI-spec endpoint support
  • Framework-agnostic: PyTorch, JAX, TensorFlow, or an external engine such as vLLM
  • Runs locally with pip install litserve and python server.py

Weaknesses

  • Not a ready-to-run LLM server; you write the serving code for each model
  • Authentication is DIY when self-hosted; token and password auth are listed for managed only
  • Load balancing, scale-to-zero and multi-node inference are listed as managed-only features
  • The 2x-faster-than-FastAPI claim comes from the project's own benchmarks

What it needs

  • GPU optional
  • Needs FastAPI
  • Models: PyTorch, JAX, TensorFlow, Hugging Face, vLLM
  • port 8000

What changed

From the release notes. No flag means the note does not say so, not that the upgrade is safe.

Also in Model serving

See all 76
Also in Model serving
RankProjectScore
1llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server135.7k starsMIT94 out of 100
2OllamaRuns open models locally through a CLI and REST API187.5k starsMITknown advisory in latest version92 out of 100
3LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs50.4k starsMIT91 out of 100
#4vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API97.1k starsApache-2.088 out of 100
#5strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM45.4k starsMIT88 out of 100