#8484th of 105 in Model serving

ik_llama.cpp

llama.cpp fork with faster CPU inference and extra quantization types

Releases and problems feed (RSS)

Stars
3.4k
License
MIT
Last commit
Nov 2026
Language
C++
Stars, last 29 weeks+1,178
29 Apr 2026: 2,231 stars6 May 2026: 2,296 stars13 May 2026: 2,390 stars20 May 2026: 2,471 stars27 May 2026: 2,555 stars3 Jun 2026: 2,641 stars10 Jun 2026: 2,681 stars17 Jun 2026: 2,710 stars24 Jun 2026: 2,751 stars1 Jul 2026: 2,794 stars8 Jul 2026: 2,822 stars15 Jul 2026: 2,868 stars22 Jul 2026: 2,915 stars29 Jul 2026: 2,938 stars5 Aug 2026: 2,976 stars12 Aug 2026: 2,995 stars19 Aug 2026: 3,031 stars26 Aug 2026: 3,060 stars2 Sept 2026: 3,141 stars9 Sept 2026: 3,176 stars16 Sept 2026: 3,202 stars23 Sept 2026: 3,226 stars30 Sept 2026: 3,247 stars7 Oct 2026: 3,265 stars14 Oct 2026: 3,286 stars21 Oct 2026: 3,307 stars28 Oct 2026: 3,339 stars4 Nov 2026: 3,367 stars11 Nov 2026: 3,387 stars18 Nov 2026: 3,409 stars

2,231 on 29 Apr 20263,409 on 18 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

A fork of llama.cpp that serves GGUF models through llama-server, with a built-in web UI and HTTP API on port 8080. It adds IQK, trellis and ternary quantization types, MLA/FlashMLA, graph-parallel multi-GPU, MoE expert caching and speculative decoding (MTP, DFlash). Only the CPU (AVX2/ARM_NEON or newer) and CUDA backends are fully supported.

Who it is for: Self-hosters running large GGUF and MoE models on CPU or NVIDIA GPUs

Strengths

  • Many extra quant types: IQK, trellis (KT), ternary, NVFP4_KS, with AVX-512 repacking
  • Hybrid CPU/GPU MoE inference with tensor overrides and a dynamic expert cache
  • Split mode graph for multi-GPU CUDA; MTP, DFlash and self-speculative decoding
  • Adds new model architectures quickly; supported list is kept in the README

Weaknesses

  • Only CPU and CUDA work well; ROCm, Vulkan and Metal issues are explicitly not handled
  • Diverged from upstream; last synced with llama.cpp in August 2024
  • -rtr can slow prompt processing in hybrid MoE setups, since k-quants lack CUDA interleaved kernels
  • Unsloth _XL quants containing f16 tensors are likely to fail; no tagged releases

What it needs

  • GPU optional
  • Docker
  • Needs CMake, CUDA Toolkit (for GPU builds)
  • Models: GGUF, DeepSeek-V3/V4, Qwen3/3.5, GLM-4.5/5, Gemma3/4
  • port 8080

Known problems

Open now, reported in the last 90 days

Reported, then fixed by a linked change

Titles are the reporters' own words, taken from the project's GitHub issues and ranked by reactions. We have not tried to reproduce any of them.

Also in Model serving

See all 105
Also in Model serving
RankProjectScore
1llama.cppC/C++ inference engine for GGUF LLMs and VLMs with an OpenAI-compatible server135.9k starsMIT94 out of 100
2OllamaRuns open models locally through a CLI and REST API187.8k starsMITknown advisory in latest version91 out of 100
3LocalAISelf-hosted model server with OpenAI, Anthropic and ElevenLabs-compatible APIs50.5k starsMIT91 out of 100
#4strataDesktop and terminal app that runs GGUF models locally with OpenAI-compatible APIs46.4k stars+16,488 (+55%) in 30 daysMIT89 out of 100
#5modularMojo language and MAX inference framework with an OpenAI-compatible server30.2k stars+228 (+0.8%) in 30 dayscustom license89 out of 100