#4747th of 57 in Model serving
serving
Inference server for TensorFlow SavedModels with gRPC and REST endpoints
- Stars
- 6.4k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- C++
6,318 on 4 Apr 20266,365 on 31 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
TensorFlow Serving loads trained models, manages their versions, and serves them over gRPC and HTTP. It serves multiple models or versions at once, supports canarying and A/B testing, and includes a batching scheduler for joint GPU execution with configurable latency limits. It targets TensorFlow SavedModels but can be extended with custom C++ servables.
Who it is for: ML engineers deploying TensorFlow models to production inference
Strengths
- Serves multiple models or versions at once; new versions need no client changes
- Exposes both gRPC and REST predict endpoints
- Batching scheduler groups requests for GPU with configurable latency controls
- Modular C++ design: custom servables and version sources can be added
Weaknesses
- Built around TensorFlow SavedModel; other model types need a custom C++ servable
- No LLM-specific features in the README, such as GGUF or an OpenAI-compatible API
- Non-Docker install is marked not recommended in the README
- Extending or building from source requires C++ and a Bazel-based build
What it needs
- GPU optional
- Needs Docker (recommended install)
- Models: TensorFlow SavedModel, embeddings, vocabularies, feature transformations
- port 8501
Quick answers
Is serving open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Can you run serving with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does serving need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Model serving
See all 57| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 91 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |