#1111th of 29 in Model serving
ds4
Native C inference engine for a few large open models on Mac, CUDA and ROCm
- Stars
- 23.7k
- License
- MIT
- Last commit
- Oct 2026
- Language
- C
Missed the latest GitHub check; these facts may be older.
Overview
DwarfStar is a narrow C inference engine for DeepSeek V4 Flash and PRO, GLM 5.x and Qwen3.8 Flash Next, with Metal, CUDA and ROCm backends. It ships a CLI, an HTTP server (default 127.0.0.1:8000), a native coding agent and an eval tool. It is not a general GGUF runner and only loads the GGUF files the project produces. SSD streaming lets models run beyond RAM.
Who it is for: Self-hosters running large open LLMs on Macs, DGX Spark or multi-GPU boxes
Strengths
- Metal, CUDA (DGX Spark, multi-GPU, Ada/L40S) and ROCm on Strix Halo backends
- SSD streaming runs models larger than RAM; tensor and pipeline parallelism across machines
- Includes server, CLI, native coding agent and ds4-eval regression suite in one repo
- Self-contained C code that does not link against GGML
Weaknesses
- Not a general GGUF runner; only project-produced GGUF files load
- README calls it beta quality, fast-changing, with possible regressions
- Model support is opportunistic; models may be removed when replaced
- No tagged releases; built from source with make
What it needs
- RAM ≥ 64 GB
- GPU optional
- Models: DeepSeek V4 Flash, DeepSeek V4.1 Flash, DeepSeek V4 PRO, GLM 5.2, GLM 5.3
- port 8000
Also in Model serving
See all 29| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 82 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| 3 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 72 out of 100 |