#1818th of 23 in Model serving
airllm
Python library that runs large LLMs on small GPUs by streaming layers from disk
- Stars
- 38.6k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Jupyter Notebook
14,067 on 24 Mar 202638,618 on 13 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
AirLLM splits a Hugging Face model into per-layer shards and keeps only one decoder layer on the GPU at a time, so VRAM need tracks layer size rather than total model size. The README reports Llama 3.x 70B in about 4GB, DeepSeek-V3 671B in about 12GB, and LoRA training of Qwen3.8-Flash-Next 125B under 6GB. Loading goes through a single AutoModel.from_pretrained call, with optional 4bit/8bit compression.
Who it is for: Engineers running or fine-tuning very large open LLMs on low-VRAM GPUs
Strengths
- Runs 70B models in about 4GB VRAM without quantization, distillation or pruning, per the README
- One AutoModel.from_pretrained call covers Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma and others
- Includes LoRA training that streams frozen weights layer by layer, with example scripts
- Supports Apple silicon via mlx and CPU inference, besides CUDA GPUs
Weaknesses
- Splitting the model needs large disk space, for example about 360GB for Flash-Next
- Speed is bound by disk loading; the README gives no tokens-per-second figures
- No HTTP API or server; it is a Python library used with generate()
- Newer models need specific transformers versions, and Kimi K3 needs flash-attn on CUDA 12
What it needs
- GPU optional
- Needs PyTorch, transformers, bitsandbytes (for 4bit/8bit compression), mlx (macOS), flash-attn and compressed-tensors (Kimi K3)
- Models: Llama 2/3/3.1/3.3/4, Qwen 1/2/2.5/3/3.5/3.8, DeepSeek V2/V3/R1, Mistral, Mixtral
Also in Model serving
See all 23| Rank | Project | Score |
|---|---|---|
| 1 | LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends | 81 out of 100 |
| 2 | llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API | 80 out of 100 |
| 3 | OllamaLocal model runner with CLI and REST API for open-weight LLMs | 79 out of 100 |
| #4 | vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching | 76 out of 100 |
| #5 | colibriC inference engine that runs huge MoE models by streaming experts from disk | 71 out of 100 |