#2222nd of 29 in Model serving
web-llm
In-browser LLM inference engine with an OpenAI-style JavaScript API
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 19.3k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- TypeScript
Missed the latest GitHub check; these facts may be older.
Overview
WebLLM is an npm package that runs MLC-format language models inside the browser using WebGPU, with no server component. It exposes an OpenAI-style engine.chat.completions interface with streaming, JSON mode and seeding, and supports Web Worker, Service Worker and Chrome extension setups. Custom models must be compiled to MLC format with MLC LLM.
Who it is for: Web developers embedding local LLM inference in browser apps or extensions
Strengths
- Runs fully client-side on WebGPU, so no inference server is needed
- OpenAI-style chat API with streaming, JSON mode and seed support
- Worker, Service Worker and Chrome extension engines, with examples for each
- Optional SRI hash verification of model config, WASM and tokenizer files
Weaknesses
- Requires a WebGPU-capable browser
- First load downloads full model weights; large models take a long time
- Function calling is marked WIP; the model request parameter is ignored
- Custom models must be compiled to MLC format through MLC LLM
What it needs
- GPU required
- Needs WebGPU-capable browser, MLC LLM (for compiling custom models)
- Models: Llama 3, Llama 2, Phi 3, Gemma-2B, Mistral-7B
Also in Model serving
See all 29| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 82 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| 3 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 72 out of 100 |