#120120th of 242 in ML frameworks and libraries
mteb
Python toolkit for benchmarking embedding models on retrieval and other tasks
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 3.5k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- Python
3,215 on 25 Apr 20263,518 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
MTEB is a Python library and CLI that evaluates text and multimodal embedding models across a catalog of tasks and named benchmarks, such as MTEB(eng, v2). Models load through mteb.get_model, which falls back to SentenceTransformer for models not implemented in MTEB. It also backs the public MTEB leaderboard on Hugging Face.
Who it is for: Engineers and researchers comparing or submitting embedding models
Strengths
- Installs with pip or uv and runs from Python or the `mteb run` CLI
- Named benchmarks carry citations and can export task tables as LaTeX
- Feeds the public Hugging Face MTEB leaderboard, with documented steps to add models
- Falls back to SentenceTransformer for models without a custom MTEB implementation
Weaknesses
- Evaluation only; it does not train, fine-tune or serve embedding models
- Task and model catalogs live in external docs, not the README
What it needs
- Docker
- Needs sentence-transformers
- Models: SentenceTransformer models, custom model definitions
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in ML frameworks and libraries
See all 242| Rank | Project | Score |
|---|---|---|
| 1 | transformersModel-definition library for pretrained text, vision, audio and multimodal models | 94 out of 100 |
| 2 | ultralyticsPython library and CLI for training and running YOLO vision models | 94 out of 100 |
| 3 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 94 out of 100 |
| #4 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |