#8484th of 105 in Model serving
ik_llama.cpp
llama.cpp fork with faster CPU inference and extra quantization types
- Stars
- 3.4k
- License
- MIT
- Last commit
- Nov 2026
- Language
- C++
2,231 on 29 Apr 20263,409 on 18 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
A fork of llama.cpp that serves GGUF models through llama-server, with a built-in web UI and HTTP API on port 8080. It adds IQK, trellis and ternary quantization types, MLA/FlashMLA, graph-parallel multi-GPU, MoE expert caching and speculative decoding (MTP, DFlash). Only the CPU (AVX2/ARM_NEON or newer) and CUDA backends are fully supported.
Who it is for: Self-hosters running large GGUF and MoE models on CPU or NVIDIA GPUs
Strengths
- Many extra quant types: IQK, trellis (KT), ternary, NVFP4_KS, with AVX-512 repacking
- Hybrid CPU/GPU MoE inference with tensor overrides and a dynamic expert cache
- Split mode graph for multi-GPU CUDA; MTP, DFlash and self-speculative decoding
- Adds new model architectures quickly; supported list is kept in the README
Weaknesses
- Only CPU and CUDA work well; ROCm, Vulkan and Metal issues are explicitly not handled
- Diverged from upstream; last synced with llama.cpp in August 2024
- -rtr can slow prompt processing in hybrid MoE setups, since k-quants lack CUDA interleaved kernels
- Unsloth _XL quants containing f16 tensors are likely to fail; no tagged releases
What it needs
- GPU optional
- Docker
- Needs CMake, CUDA Toolkit (for GPU builds)
- Models: GGUF, DeepSeek-V3/V4, Qwen3/3.5, GLM-4.5/5, Gemma3/4
- port 8080
Known problems
Open now, reported in the last 90 days
- MoE expert offload: decode throughput tracks host bytes per token, not VRAM occupancy ↗ (opens in a new tab)1 reactions
- KVarN: structured KV-cache quantization (WIP — honest status, seeking direction) ↗ (opens in a new tab)1 reactions
Reported, then fixed by a linked change
- Bug: Qwen3.5 multimodal not working with --split-mode graph ↗ (opens in a new tab)2 reactions · closed 10 Mar 2026Fixed by Make split mode graph work with vision enabled ↗ (opens in a new tab)
- Bug: Server crashes on /completions request after /chat/completions when using --jinja flag ↗ (opens in a new tab)1 reactions · closed 17 May 2026Fixed by fix(server): reset chat parser on slot reuse to prevent crash (#1763) ↗ (opens in a new tab)
- Bug: -ts not working since #1501 (even without --fit) ↗ (opens in a new tab)1 reactions · closed 28 Mar 2026Fixed by Honor manual splits ↗ (opens in a new tab)
- Bug: ik_llama.cpp crashes when Kimi K2 Thinking sometimes produces invalid tool call due to not working grammar ↗ (opens in a new tab)1 reactions · closed 30 Nov 2025Fixed by Update grammar ↗ (opens in a new tab)
Titles are the reporters' own words, taken from the project's GitHub issues and ranked by reactions. We have not tried to reproduce any of them.
Also in Model serving
See all 105| Rank | Project | Score |
|---|---|---|
| 1 | llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server | 94 out of 100 |
| 2 | OllamaRuns open models locally through a CLI and REST API | 91 out of 100 |
| 3 | LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs | 91 out of 100 |
| #4 | strataDesktop and terminal app that runs GGUF models locally with OpenAI-compatible APIs | 89 out of 100 |
| #5 | modularMojo language and MAX inference framework with an OpenAI-compatible server | 89 out of 100 |