#6666th of 105 in Model serving
project-maya
Runs very large MoE LLMs locally by streaming experts between GPU, RAM and SSD
- Stars
- 3.1k
- License
- MIT
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- C++
203 on 8 Oct 20263,134 on 19 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Project Maya is an inference engine with custom GGUF quants for GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash, MiniMax-M3.1-Flash and Hy4-Flash. It keeps frequently used experts in VRAM, the next ones in RAM and the rest on NVMe, tuning itself to the measured hardware. It serves an OpenAI- and Anthropic-compatible API with tool calls, plus a browser chat, a live monitor and a Models page.
Who it is for: Self-hosters running 200B+ open LLMs on one or a few GPUs
Strengths
- Tiered expert placement across VRAM, RAM and SSD lets 200-750B models run on one GPU
- Publishes user benchmarks with hardware and version, e.g. 81 tokens/s on one RTX PRO 6000
- OpenAI- and Anthropic-compatible API with tool calls; up to 8 concurrent requests on three models
- Runs on NVIDIA from V100/RTX 20, AMD and Windows, with a one-command setup
Weaknesses
- Needs roughly 56 GB to 192 GB of RAM plus VRAM, depending on the model
- Each model needs its own engine; support is limited to the five listed families
- Mixed licenses: GLM-5.3, MiniMax and Hy4 models carry their own non-MIT terms
- Quants differ from originals, e.g. 83-90% next-token match on GLM-5.3-Flash
What it needs
- RAM ≥ 56 GB
- GPU required
- Docker
- Needs NVMe SSD
- Models: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, MiniMax-M3.1-Flash, Hy4-Flash
- port 8080
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Known problems
Open now, reported in the last 90 days
- MCP servers as a way to make models smarter without more VRAM, and why Maya could treat each server's usage guide as a skill ↗ (opens in a new tab)1 reactions
- Speed report: 1× RTX 4090 48 GB, DeepSeek-V4.1-Flash-1017 — Maya-S IQ2_XXS vs 2.25 bpw, 18-batch code review (v1.1.65) ↗ (opens in a new tab)1 reactions
- [Bug] Engine build fails on Windows: cudafe++ crashed with status 0xC0000005 using CUDA 13.3 and VS 2026 (MSVC 14.52) ↗ (opens in a new tab)1 reactions
- Windows: CUDA page-locking caps the RAM tier at ~90 GB per process - RAM-resident mode cannot start (32 GB card, 192 GB RAM, 128K ctx) ↗ (opens in a new tab)1 reactions
Reported, then fixed by a linked change
- ds4: MAYA_DS4_TP_GGUF_UNEVEN=1 with --tp 2 on Maya-S fails on the first streamed expert ("streaming layer 1 expert 374 failed") ↗ (opens in a new tab)0 reactions · closed 24 Oct 2026Fixed by ds4: DS4_FAT_ROWS's buffers sized for the widest FFN a device's layers give it (#439) ↗ (opens in a new tab)
Titles are the reporters' own words, taken from the project's GitHub issues and ranked by reactions. We have not tried to reproduce any of them.
Also in Model serving
See all 105| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaRuns open models locally through a CLI and REST API | 91 out of 100 |
| 3 | LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs | 91 out of 100 |
| #4 | strataDesktop and terminal app that runs GGUF models locally with OpenAI-compatible APIs | 89 out of 100 |
| #5 | modularMojo language and MAX inference framework with an OpenAI-compatible server | 89 out of 100 |