#4646th of 65 in Model serving
mesh-llm
Pools GPUs across machines behind one OpenAI-compatible API
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.1k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- Rust
834 on 21 Apr 20264,063 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Mesh LLM joins machines into a mesh and serves GGUF models through a single OpenAI-compatible API on port 9337. Requests route by the model field to a peer that can serve it. Models too large for one node are split by layer across nodes using the Skippy runtime, a patched llama.cpp. Peer traffic is encrypted over QUIC, and a web console runs on port 3131.
Who it is for: Self-hosters pooling several GPU machines to run large local models
Strengths
- Splits models too large for one machine into layer stages across peers
- Same /v1 endpoint on every node, routed by model name
- Prebuilt bundles for macOS Metal, plus Linux and Windows on CUDA, ROCm, Vulkan, CPU
- Private meshes via invite token, or public discovery over Nostr
Weaknesses
- Mixture-of-Agents mode (model: mesh) is experimental and needs at least two models
- Native Windows CUDA runtime does not cover RTX 50-series (Blackwell) yet
- Local-only mode needs absolute, non-symlink GGUF paths and skips mesh features
- Source build needs just, cmake, Rust, sccache, mold or lld, and Node.js 24
What it needs
- GPU optional
- Models: GGUF, Qwen, Llama, Gemma, Mistral
- port 3131
Quick answers
Is mesh-llm open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Does mesh-llm need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Model serving
See all 65| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 91 out of 100 |
| #4 | vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API | 88 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 88 out of 100 |