#8282nd of 126 in ML frameworks and libraries
flash-linear-attention
Triton kernels and PyTorch layers for linear attention and hybrid sequence models
- Stars
- 6k
- License
- MIT
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- Python
4,759 on 3 Apr 20266,010 on 30 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Library of Triton kernels, `fla.layers` token-mixing modules and model implementations for linear attention, sparse attention, state space models and hybrids. It covers RetNet, GLA, DeltaNet, Gated DeltaNet, KDA, RWKV7, Mamba2/3, NSA and MoBA, and the layers can replace standard multihead attention. The README states the implementations are verified on NVIDIA, AMD and Intel hardware.
Who it is for: Researchers and engineers training or benchmarking subquadratic LLMs
Strengths
- Covers about 40 architectures, from RetNet and GLA to Mamba3, KDA and Gated DeltaNet
- Layers can replace standard attention in an existing model, with hybrid model support
- Backend extras for CUDA, ROCm, XPU, NPU and CPU installs
- Variable-length kernels, fused linear cross-entropy, context parallel for KDA and GDN
Weaknesses
- Bare pip install does not pull torch or triton; a backend extra is required
- Full training loop lives in a separate repo (flame), not here
- Input layout changed from head-first to seq-first in 2024, which can break older code
What it needs
- GPU optional
- Needs PyTorch, Triton
- Models: RetNet, GLA, DeltaNet, Gated DeltaNet, KDA
Quick answers
Is flash-linear-attention open source?
Yes. Its code is on GitHub under the MIT license.
Does flash-linear-attention need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in ML frameworks and libraries
See all 126| Rank | Project | Score |
|---|---|---|
| 1 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 95 out of 100 |
| 2 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 91 out of 100 |