#130130th of 206 in ML frameworks and libraries
lmms-eval
Evaluation harness for multimodal models across image, video and audio benchmarks
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.5k
- License
- custom license
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
lmms-eval is a Python evaluation harness that runs 100+ benchmarks (MMMU, MME, MMBench, VideoMME and others) against multimodal models over image, video and audio inputs. It supports 30+ model families plus vLLM, SGLang, Ollama and OpenAI-compatible endpoints, and reports confidence intervals and paired comparisons. It also ships a FastAPI-style HTTP eval server with job queue and an optional Web UI.
Who it is for: Model developers benchmarking multimodal LLMs on image, video and audio tasks
Strengths
- 100+ tasks across image, video and audio, with YAML task definitions for custom datasets
- Runs models via vLLM, SGLang, Ollama, OpenAI-compatible APIs or native wrappers
- Reports confidence intervals, clustered standard errors and paired t-tests, not just accuracy
- HTTP eval server with job queue lets training loops submit checkpoint evaluations asynchronously
Weaknesses
- Eval server has no authentication; README says trusted networks only
- Caption tasks (coco, refcoco, nocaps) need Java 1.8 installed
- Web UI requires Node.js 18+ and builds the frontend on first run
- License detected as NOASSERTION, so terms need manual checking
What it needs
- GPU optional
- Needs uv, Python, Node.js 18+ (Web UI), Java 1.8 (caption tasks), Hugging Face datasets, OpenAI API key (some tasks)
- Models: Qwen2.5-VL, Qwen3-VL, Qwen3.5, LLaVA-OneVision, InternVL-2
- port 8000
Also in ML frameworks and libraries
See all 206| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | ultralyticsPython library and CLI for training and running YOLO vision models | 94 out of 100 |
| 3 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |