#7979th of 168 in ML frameworks and libraries
transformerengine
NVIDIA library for FP8 and FP4 Transformer training in PyTorch and JAX
- Stars
- 3.6k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- Python
3,259 on 21 Apr 20263,602 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Transformer Engine provides fused kernels and drop-in modules for attention, linear layers, normalization and MoE, with autocast-style APIs for PyTorch and JAX. It manages scaling factors and amax histories for FP8, MXFP8 and NVFP4 recipes, so low-precision training can be added without rewriting a training stack. It also supports tensor, sequence, context and expert parallelism with communication overlap.
Who it is for: Engineers training large Transformer or MoE models on NVIDIA GPUs
Strengths
- Handles FP8 scaling factors and amax histories automatically via configurable recipes
- Supports both PyTorch and JAX with autocast-style contexts
- FP8 on Hopper, Ada, Blackwell, Rubin; MXFP8 and NVFP4 on Blackwell
- Integrated with Megatron-LM, Hugging Face Accelerate, Lightning and NeMo Megatron Bridge
Weaknesses
- NVIDIA GPUs only; low-precision modes depend on specific GPU generations
- Training-focused; the README describes no inference or serving path
- Install needs --no-build-isolation; source builds have extra prerequisites
- No Docker image in the repo; ready-to-run setup relies on NGC containers
What it needs
- GPU required
- Needs PyTorch or JAX, NVIDIA CUDA
- Models: Transformer models, MoE, FP8, MXFP8, NVFP4
Quick answers
Is transformerengine open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Does transformerengine need a GPU?
Yes, it needs a GPU.
Also in ML frameworks and libraries
See all 168| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |