#4646th of 57 in Model serving
shimmy
OpenAI-compatible inference server for GGUF models, written in Rust on WebGPU
- Stars
- 6k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Rust
3,824 on 3 Apr 20266,004 on 30 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Shimmy is a single-binary server that exposes GGUF models through OpenAI-compatible chat completions, text completions, streaming and model endpoints. It runs on Airframe, a pure-Rust WebGPU (WGSL) engine, so it works on NVIDIA, AMD, Intel and Apple GPUs without a C++ toolchain or Python. 26 model/quant combinations across 12 families are certified for math, inference and determinism.
Who it is for: Self-hosters wanting a lightweight local OpenAI-compatible GGUF server
Strengths
- Pure Rust with WebGPU shaders; no C++ toolchain, Python runtime or backend flags
- Deterministic output for the same model, seed and params
- Runs on NVIDIA, AMD, Intel and Apple Silicon GPUs through one backend
- Optional INT4 KV cache claims about 7x lower KV memory in tested configurations
Weaknesses
- v2 removed llama.cpp, MLX and HuggingFace backends; only Airframe remains
- Only 26 certified model/quant combinations; mostly Q4_K_M quants
- SafeTensors native inference and MoE CPU offload are still roadmap items
- README lists MIT while detected license is Apache-2.0; terms need checking
What it needs
- GPU required
- Docker + Compose
- Models: GGUF, Llama, Qwen2, Qwen3, Qwen3.5
- port 11435
Quick answers
Is shimmy open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Can you run shimmy with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does shimmy need a GPU?
Yes, it needs a GPU.
Also in Model serving
See all 57| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 93 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 91 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 90 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 88 out of 100 |
| #5 | colibriC inference engine that streams MoE experts from disk to run huge models | 85 out of 100 |