#2121st of 22 in Model serving
bitnet
Inference framework for 1.58-bit ternary LLMs on CPU and GPU
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 41.1k
- License
- MIT
- Last commit
- Jul 2026
- Language
- C++
35,953 on 24 Mar 202641,139 on 13 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
bitnet.cpp is Microsoft's llama.cpp-based inference framework with optimized I2_S, TL1 and TL2 kernels for ternary BitNet b1.58 models on x86 and ARM CPUs, plus a GPU kernel. It runs GGUF models through Python scripts for setup, chat and benchmarking. The README reports 1.37x to 6.17x CPU speedups over full-precision baselines.
Who it is for: Engineers running or benchmarking 1-bit LLMs on CPUs or edge devices
Strengths
- Reports 2.37x-6.17x speedups on x86 and 1.37x-5.07x on ARM versus full precision
- Runs a 100B BitNet b1.58 model on one CPU at 5-7 tokens per second (per README)
- Supports official 2B-4T and embedding models plus Falcon3, Falcon-E and Llama3-8B ternary variants
- Includes an end-to-end benchmark script and a dummy-model generator
Weaknesses
- Only ternary 1-bit models are supported; ordinary GGUF models are out of scope
- No HTTP server or OpenAI-compatible API described; usage is via run_inference.py CLI
- Kernel support is uneven across models; embedding models have no ARM support
- Build needs clang>=18, cmake>=3.22 and Python>=3.10; no Docker or release builds
What it needs
- GPU optional
- Needs python>=3.10, cmake>=3.22, clang>=18, conda, huggingface-cli
- Models: BitNet-b1.58-2B-4T, BitNet-embedding-0.6B, BitNet-embedding-270M, bitnet_b1_58-large, bitnet_b1_58-3B
Also in Model serving
See all 22| Rank | Project | Score |
|---|---|---|
| 1 | LocalAIOne OpenAI-compatible server for text, speech, image and video models | 81 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 80 out of 100 |
| 3 | OllamaRuns open-weight models locally behind a CLI and REST API | 79 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 71 out of 100 |