ArchestackBest of Open-Source AI: List on GitHub ↗3 stars (opens in a new tab)

#1010th of 29 in Model serving

strata

Runs a 125B-parameter MoE model on one consumer GPU plus system RAM

Repository on GitHub ↗ (opens in a new tab)
Stars
22.7k
License
MIT
Last commit
Oct 2026
Last release
Oct 2026
Language
C++

Missed the latest GitHub check; these facts may be older.

Overview

Strata is a C++ inference engine, built on parts of llama.cpp/ggml, that runs Qwen3.8-Flash-Next on a PC with a 12 GB or larger NVIDIA or AMD card. It keeps the most-used experts in VRAM, holds all of them in RAM and uses the CPU for the rest. It serves a browser chat app plus OpenAI-, Anthropic- and Responses-style APIs on port 8080.

Who it is for: Self-hosters with a 12 GB+ gaming GPU and 32 GB+ RAM wanting a large local model

Strengths

  • Serves OpenAI /v1, Anthropic /v1/messages and Responses /v1/responses on 127.0.0.1:8080
  • Reports 53-94 tokens/s decode on an RTX 5070 12 GB, depending on quantization
  • Installer picks a model size by RAM; interrupted downloads resume
  • Ships a BENCH script, an MCP server for setup, and multi-GPU support

Weaknesses

  • Built around one model family (Qwen3.8-Flash-Next), not arbitrary GGUFs
  • Needs 32 GB or more RAM and about 80 GB disk; PC may freeze 1-3 minutes at load
  • Answers one request at a time by default; parallel mode slows each answer on 12 GB cards
  • Intel Arc, Mac, older GPUs and Strix Halo support is experimental and community-tested

What it needs

  • RAM ≥ 32 GB
  • GPU required
  • Docker
  • Needs NVIDIA or AMD GPU driver, Qwen3.8-Flash-Next model download (about 70 GB)
  • Models: Qwen3.8-Flash-Next GGUF (Q2_0, IQ2_XS, IQ3_XXS, IQ3_S, Coder, Swift 1.5), Unsloth UD-IQ4_XS to Q8_0
  • port 8080

Also in Model serving

See all 29
Also in Model serving
RankProjectScore
1LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends49.5k starsMIT82 out of 100
2OllamaLocal model runner with CLI and REST API for open-weight LLMs183.2k starsMIT80 out of 100
3llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API131.2k starsMIT80 out of 100
#4vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching93.8k starsApache-2.076 out of 100
#5colibriC inference engine that runs huge MoE models by streaming experts from disk41.9k starsApache-2.072 out of 100