ArchestackBest of Open-Source AI: List on GitHub ↗ (opens in a new tab)

#2727th of 65 in Model serving

edge0

Streaming MoE inference that serves 35B models from SSD on Apple Silicon

Website ↗ (opens in a new tab)Repository on GitHub ↗ (opens in a new tab)
Stars
9.7k
License
Apache-2.0
Last commit
Nov 2026
Last release
Nov 2026
Language
C++
Stars, last 9 weeks+9,177
7 Sept 2026: 520 stars14 Sept 2026: 2,215 stars21 Sept 2026: 2,537 stars28 Sept 2026: 2,628 stars5 Oct 2026: 3,661 stars12 Oct 2026: 4,803 stars19 Oct 2026: 5,958 stars26 Oct 2026: 7,504 stars2 Nov 2026: 9,458 stars9 Nov 2026: 9,697 stars

520 on 7 Sept 20269,697 on 9 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

edge0 streams mixture-of-experts weights from SSD on demand, so peak memory tracks the active expert set rather than total parameters. It pairs int4 checkpoints with Recover-LoRA adapters and a trained prerouter that predicts expert routing one step ahead. The Python/MLX framework runs on Apple Silicon and serves an OpenAI-compatible /v1/chat/completions endpoint; macOS, iOS, Android and Windows engines live in the same repo.

Who it is for: Mac and mobile users running large MoE models locally on limited memory

Strengths

  • Reported 2.9 GiB peak MLX allocator memory for the 35B tier on a Mac mini M4 Pro
  • Ships trained LoRA and prerouter adapters with the checkpoints, ready to serve
  • Separate engines for macOS (Rust), iOS (Swift), Android (Kotlin) and Windows (C++/Vulkan)
  • Self-run OpenCompass results: int4 averages 3.9 points below fp16 for the 35B tier

Weaknesses

  • Python framework supports only macOS on Apple Silicon; CUDA backend is roadmap only
  • Decode speed is 14.9-17.7 tok/s for the 35B tier, 23.9-25.3 for the 8B
  • Only two preview model tiers, built on Qwen3.6-35B-A3B and Ling 3.0
  • Unified cross-platform framework is unreleased; platform engines use separate stacks

What it needs

  • GPU required
  • Needs MLX 0.30.6, mlx-lm 0.31.0, Python 3.10+
  • Models: Edge0-35B-A3B-preview, Edge0-8B-A1B-preview, int4 safetensors
  • port 8000

What changed

  1. v0.2.0 ↗ (opens in a new tab)
  2. v0.1.1 ↗ (opens in a new tab)
  3. v0.1.0 ↗ (opens in a new tab)

From the release notes. No flag means the note does not say so, not that the upgrade is safe.

Quick answers

Is edge0 open source?

Yes. Its code is on GitHub under the Apache-2.0 license.

Does edge0 need a GPU?

Yes, it needs a GPU.

Also in Model serving

See all 65
Also in Model serving
RankProjectScore
1llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server134.1k stars+3,480 (+2.7%) in 30 daysMIT94 out of 100
2OllamaLocal model runner with CLI and REST API for open-weight LLMs186.7k stars+4,310 (+2.4%) in 30 daysMITknown advisory in latest version92 out of 100
3LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends50.3k stars+940 (+1.9%) in 30 daysMIT91 out of 100
#4vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API96.7k stars+3,268 (+3.5%) in 30 daysApache-2.088 out of 100
#5strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM43.1k stars+21,265 (+97%) in 30 daysMIT88 out of 100