#2222nd of 26 in ML frameworks and libraries
flash-attention
Fused GPU kernels for exact attention in PyTorch, FlashAttention 2 through 4
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 25.1k
- License
- BSD-3-Clause
- Last commit
- Oct 2026
- Last release
- Jun 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Official implementation of FlashAttention and FlashAttention-2 as CUDA and ROCm kernels exposed through a PyTorch Python API (flash_attn_func, varlen, and KV-cache variants). The repo also ships FlashAttention-3 (beta, Hopper) and FlashAttention-4 (CuTeDSL, Hopper and Blackwell). Supports causal masks, sliding window, ALiBi, MQA/GQA, and paged KV cache.
Who it is for: ML engineers writing or optimizing transformer training and inference code
Strengths
- Official reference implementation with papers for FA1, FA2 and FA3
- Covers NVIDIA (Ampere, Ada, Hopper, Blackwell via FA4) and AMD ROCm (CK and Triton backends)
- Decoding-oriented API: in-place KV cache update, rotary, paged cache, GQA/MQA
- Head dimensions up to 256 with fp16 and bf16; FP8 forward in FA3
Weaknesses
- Source build is slow without ninja: about 2h versus 3-5 minutes on 64 cores
- FA3 is a beta, requires H100/H800 and CUDA 12.3 or newer
- Linux only; Windows support is untested and unconfirmed
- flash_attn_with_kvcache has no backward pass; Turing GPUs need a separate repo
What it needs
- GPU required
- Needs PyTorch 2.2+, CUDA 12.0+ or ROCm 6.0+, ninja, packaging, psutil
- Models: fp16, bf16, fp8 (FA3 forward), fp32 (ROCm Triton)
Also in ML frameworks and libraries
See all 26| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 93 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 89 out of 100 |
| #4 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 85 out of 100 |
| #5 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 83 out of 100 |