#4343rd of 47 in Model serving
bentoml
Python library for turning model inference code into batched REST APIs
- Stars
- 8.9k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- May 2026
- Language
- Python
8,498 on 1 Apr 20268,914 on 21 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
BentoML is a Python library that wraps model inference code in a service class with typed API methods and serves it over HTTP, on port 3000 by default. It packages code, models and dependencies into a Bento artifact and can generate a Docker image from it. Serving features include adaptive batching, multi-model composition and GPU inference.
Who it is for: Python engineers deploying custom or open models as production inference APIs
Strengths
- Any ML framework or modality works, since the service is plain Python with type hints
- Built-in adaptive batching, worker parallelism and multi-model composition
- bentoml build and containerize generate a reproducible Docker image from one config
- Example repos cover LLMs (vLLM), diffusion, audio, embeddings and vision
Weaknesses
- You write the service code yourself; no ready-made model server out of the box
- Requires Python 3.9 or newer
- Collects anonymous usage data by default; opt out via flag or environment variable
- Docker is needed for containerized deploys; no compose file in the repo
What it needs
- GPU optional
- Needs Python 3.9+, Docker (for containerize)
- Models: PyTorch, Transformers, vLLM, Stable Diffusion, any Python ML framework
- port 3000
Also in Model serving
See all 47| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 83 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 81 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| #4 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 79 out of 100 |
| #5 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 77 out of 100 |