#4545th of 57 in Model serving
litert-lm
On-device LLM runtime with CLI and APIs for Android, iOS, web and desktop
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 6.8k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- C++
1,243 on 4 Apr 20266,797 on 31 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
LiteRT-LM is Google's C++ orchestration layer for running LLMs on top of LiteRT on phones, browsers, desktops and Raspberry Pi. It ships a CLI (run, serve, import, describe) and language bindings, with GPU and NPU acceleration, vision and audio input, and function calling. It powers on-device GenAI in Chrome, Chromebook Plus and Pixel Watch.
Who it is for: Developers shipping local LLM features in mobile, web or embedded apps
Strengths
- Python, Kotlin and C++ APIs marked stable; Swift, JavaScript and Flutter also available
- Runs on Android, iOS, web, desktop and Raspberry Pi with GPU and NPU backends
- CLI serve exposes an OpenAI-compatible /v1/embeddings endpoint for embedding models
- Supports vision and audio input, function calling and speculative decoding
Weaknesses
- Models use the .litertlm format, so other formats likely need conversion
- Swift and JavaScript APIs are early preview; Flutter is community-maintained
- No Docker image or compose file; source build is a separate guide
What it needs
- GPU optional
- Needs LiteRT, uv (for CLI install)
- Models: Gemma, Llama, Phi-4, Qwen, EmbeddingGemma 2
Quick answers
Is litert-lm open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Does litert-lm need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Model serving
See all 57| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 91 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |