#4444th of 126 in ML frameworks and libraries
flashinfer
CUDA kernel library for LLM attention, GEMM, MoE and sampling
- Stars
- 6.8k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- Cuda
5,231 on 4 Apr 20266,828 on 31 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
FlashInfer is a Python and CUDA library that provides GPU kernels for LLM inference: paged and ragged KV-cache attention, FP8/FP4 GEMM, fused MoE, and sorting-free Top-K/Top-P sampling. It selects among backends such as FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM, and runs on Turing through Blackwell GPUs. It is a building block for inference engines; the README lists SGLang, vLLM, TensorRT-LLM and TGI among adopters.
Who it is for: Engineers building or optimizing LLM inference engines on NVIDIA GPUs
Strengths
- Covers attention, GEMM, MoE, sampling, RoPE and norm kernels behind one Python API
- Supports SM 7.5 to SM 12.1, including FP8 and NVFP4/MXFP4 on Blackwell
- Compatible with CUDAGraph and torch.compile for low-latency serving
- Used by SGLang, vLLM, TensorRT-LLM, TGI and MLC-LLM per the README
Weaknesses
- Not a server or end-user tool; needs an inference engine or custom PyTorch code
- NVIDIA GPUs only; not all features work on all compute capabilities
- Kernels compile or download on first use unless cubin and JIT-cache wheels are installed
- CUDA support limited to 12.9, 13.0 and 13.4; 13.4 runtime CI not covered
What it needs
- GPU required
- Docker
- Needs PyTorch, CUDA 12.9 or 13.0
- Models: DeepSeek MLA, Llama-4 MoE routing, DeepSeek-V3 MoE routing, LLaMA 3.1 RoPE
Quick answers
Is flashinfer open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Can you run flashinfer with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does flashinfer need a GPU?
Yes, it needs a GPU.
Also in ML frameworks and libraries
See all 126| Rank | Project | Score |
|---|---|---|
| 1 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 95 out of 100 |
| 2 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 91 out of 100 |