#5151st of 53 in Training and fine-tuning
modernbert
Research code for pre-training and evaluating ModernBERT encoder models
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 1.8k
- License
- Apache-2.0
- Last commit
- Mar 2026
- Language
- Python
1,647 on 21 Apr 20261,781 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Research repository behind ModernBERT, a BERT-style bidirectional encoder. It provides FlexBERT, a modular set of encoder building blocks configured through YAML, plus pre-training on the Composer framework and GLUE evaluation scripts. Example scripts train dense retrievers with Sentence Transformers and ColBERT models with PyLate.
Who it is for: ML researchers pre-training or evaluating encoder models on their own GPUs
Strengths
- Full pre-training and GLUE evaluation code, stated to be the code used in the paper
- YAML-configured FlexBERT blocks with example configs in the yamls folder
- Example scripts for Sentence Transformers dense retrieval and PyLate ColBERT training
- Supports Flash Attention 3 on H100 with Flash Attention 2 fallback
Weaknesses
- README describes itself as barebones and under construction
- Intermediate checkpoints are listed as release pending
- Setup needs a GPU machine, conda environment and flash-attn 2.6.3 build or wheel
- Streaming dataset class distributes memory unevenly across accelerators, per the README
What it needs
- GPU required
- Needs conda, composer, flash-attn 2.6.3, Sentence Transformers, PyLate
- Models: ModernBERT, FlexBERT, BERT (GLUE evals), ColBERT (PyLate)
Quick answers
Is modernbert open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Does modernbert need a GPU?
Yes, it needs a GPU.
Also in Training and fine-tuning
See all 53| Rank | Project | Score |
|---|---|---|
| 1 | unslothDesktop and web app to run and fine-tune local models | 88 out of 100 |
| 2 | peftPython library for LoRA and other parameter-efficient fine-tuning of Transformers models | 88 out of 100 |
| 3 | deepspeedPyTorch library for distributed training of very large models with ZeRO | 86 out of 100 |
| #4 | verlReinforcement learning training library for LLM post-training with PPO and GRPO | 86 out of 100 |
| #5 | trlPost-training library with SFT, DPO, GRPO and KTO trainers | 86 out of 100 |