#6464th of 75 in ML frameworks and libraries
xformers
PyTorch library of memory-efficient attention and fused transformer kernels
- Stars
- 10.6k
- License
- custom license
- Last commit
- Oct 2026
- Last release
- Feb 2026
- Language
- Python
10,321 on 31 Mar 202610,569 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
xFormers is a PyTorch library of transformer building blocks for research, including exact memory-efficient attention (xformers.ops.memory_efficient_attention), sparse and block-sparse attention, and fused softmax, linear, layer norm, dropout and SwiGLU ops. It ships its own CUDA kernels and dispatches to other libraries such as Flash-Attention where relevant. Wheels are published for CUDA 12.6, 12.8 and 13.0, with an experimental ROCm 7.1 build on Linux.
Who it is for: Researchers and ML engineers optimizing PyTorch transformers
Strengths
- Exact (not approximate) memory-efficient attention via a single function call
- Fused ops include softmax, layer norm, dropout(activation(x+bias)) and SwiGLU
- Prebuilt pip wheels for CUDA 12.6, 12.8 and 13.0 on Linux and Windows
- Includes python -m xformers.info to list which kernels are built
Weaknesses
- Tied to a specific PyTorch version (2.10.0 for the current stable wheels)
- Source builds can take dozens of minutes and may run out of memory
- ROCm support is marked experimental and Linux only
- License is BSD-style but detected as NOASSERTION; includes code from triton-lang/kernels
What it needs
- GPU required
- Needs PyTorch 2.10.0, CUDA 12.6/12.8/13.0 or ROCm 7.1, ninja (source builds)
- Models: PyTorch transformer models
Also in ML frameworks and libraries
See all 75| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 92 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 91 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 90 out of 100 |