#6363rd of 73 in Model serving
zml
Zig and MLIR inference stack compiling models for GPUs, TPUs and Trainium
- Stars
- 4.2k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Apr 2026
- Language
- Zig
Missed the latest GitHub check; these facts may be older.
Overview
ZML is an inference stack written in Zig on top of MLIR and built with Bazel. Models are written in Zig with tagged tensors and compiled for CUDA, ROCm, Intel OneAPI, Google TPU and AWS Trainium/Inferentia 2 from one codebase. The bundled LLM example runs Llama 3.1/3.2, Qwen 3.5 and LFM 2.5 from Hugging Face, local directories or S3.
Who it is for: Engineers deploying inference across NVIDIA, AMD, TPU and Trainium hardware
Strengths
- One Zig codebase targets CUDA, ROCm, OneAPI, TPU and Trainium via build flags
- Loads weights from hf://, local paths, https:// and s3:// through a VFS layer
- Bazel is the only prerequisite; examples run with a single bazel run command
- Includes sharding, profiler and benchmark examples for multi-device use
Weaknesses
- LLM example supports only Llama 3.1/3.2, Qwen 3.5 and LFM 2.5
- Models must be written in Zig; much smaller ecosystem than PyTorch
- Examples are CLIs; README shows no OpenAI-compatible HTTP server
- Bazel build adds setup and build-time overhead for casual use
What it needs
- GPU optional
- Needs Bazel (via bazelisk), Hugging Face token for gated models
- Models: Llama 3.1, Llama 3.2, Qwen 3.5, LFM 2.5, MNIST
Also in Model serving
See all 73| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 92 out of 100 |
| 3 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 91 out of 100 |
| #4 | vLLMInference engine serving LLMs with PagedAttention and an OpenAI-compatible API | 88 out of 100 |
| #5 | strataRuns a 125B-parameter MoE model on one consumer GPU plus system RAM | 88 out of 100 |