#1616th of 29 in Model serving
omlx
MLX-based LLM inference server for Apple Silicon with a menu bar app
- Stars
- 22.6k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
oMLX serves MLX-format LLMs, VLMs, OCR, embedding and reranker models on Apple Silicon through OpenAI- and Anthropic-compatible endpoints on port 8000. It adds continuous batching on mlx-lm and a KV cache split between RAM and SSD that survives restarts. A native SwiftUI menu bar app and a web admin dashboard handle model loading, pinning, TTL and benchmarks.
Who it is for: Mac users running local LLMs for coding agents and API clients
Strengths
- Hot RAM plus cold SSD KV cache reuses prefixes, even after a server restart
- Serves OpenAI and Anthropic APIs plus embeddings, rerank and tokenize endpoints
- Multi-model pool with LRU eviction, pinning, per-model TTL and a memory limit
- One-click setup for coding agents such as OpenCode, Codex and Hermes Agent
Weaknesses
- Apple Silicon and macOS 15+ only; no Linux, Windows or CUDA support
- Some model families need native kernels, which require full Xcode to build
- Models must be in MLX format; GGUF is not listed as supported
- Multi-Mac distributed inference is experimental and source-build only
What it needs
- GPU required
- Needs macOS 15+, Apple Silicon, Python 3.11-3.13, Xcode (optional, for native kernels)
- Models: MLX LLMs (mlx-lm), VLMs (mlx-vlm), OCR, Embeddings, Rerankers
- port 8000
Also in Model serving
See all 29| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 82 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 80 out of 100 |
| 3 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 80 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 72 out of 100 |