ArchestackBest of Open-Source AI: List on GitHub ↗3 stars (opens in a new tab)

#1313th of 29 in Model serving

mlc-llm

ML compiler and inference engine for running LLMs on GPUs, phones and browsers

Documentation ↗ (opens in a new tab)Website ↗ (opens in a new tab)Repository on GitHub ↗ (opens in a new tab)
Stars
23.2k
License
Apache-2.0
Last commit
Oct 2026
Language
Python

Missed the latest GitHub check; these facts may be older.

Overview

MLC LLM compiles language models and runs them through MLCEngine, a single inference engine targeting CUDA, ROCm, Vulkan, Metal, OpenCL and WebGPU/WASM. MLCEngine exposes an OpenAI-compatible API via a REST server and Python, JavaScript, iOS and Android bindings. The README is brief and defers install and usage details to the external docs.

Who it is for: Engineers deploying LLMs across diverse GPUs, mobile and browser targets

Strengths

  • One engine covers AMD, NVIDIA, Apple and Intel GPUs on Linux, Windows and macOS
  • Also targets browsers (WebGPU/WASM), iOS/iPadOS and Android (Adreno, Mali via OpenCL)
  • OpenAI-compatible API over REST, Python, JavaScript, iOS and Android
  • Apache-2.0 license

Weaknesses

  • README gives no install steps, model list or memory requirements; docs are external
  • No Docker or compose files detected
  • No tagged release listed

What it needs

  • GPU optional

Also in Model serving

See all 29
Also in Model serving
RankProjectScore
1LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends49.5k starsMIT82 out of 100
2OllamaLocal model runner with CLI and REST API for open-weight LLMs183.2k starsMIT80 out of 100
3llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API131.2k starsMIT80 out of 100
#4vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching93.8k starsApache-2.076 out of 100
#5colibriC inference engine that runs huge MoE models by streaming experts from disk41.9k starsApache-2.072 out of 100