#166166th of 206 in ML frameworks and libraries
vlmevalkit
Evaluation toolkit that runs vision-language models on 70+ image and video benchmarks
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.5k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Mar 2025
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
VLMEvalKit (Python package vlmeval) runs large vision-language models through benchmark suites with one command, handling dataset download, preprocessing, inference and scoring. It uses generation-based evaluation and reports both exact-match and LLM-based answer extraction results. It covers 200+ models, including commercial APIs, and backs the OpenCompass multimodal leaderboards.
Who it is for: Researchers and VLM developers benchmarking multimodal models
Strengths
- Covers 70+ image and video benchmarks and 200+ models behind one command
- Adding a custom model needs only a single generate_inner() function
- Multi-node inference via LMDeploy or vLLM for large or thinking models
- Reports both exact-match and LLM-extracted scores for comparison
Weaknesses
- Models need different pinned transformers versions, from 4.33.0 to 5.2+, so several environments are needed
- Scores may not match original benchmark papers because prompts are shared across models
- Last tagged release is March 2025, while the main branch keeps changing
- Full model and benchmark lists live in an external Feishu page, not the repo
What it needs
- GPU optional
- Needs transformers, torchvision, LMDeploy (optional), vLLM (optional), flash-attn (some models)
- Models: Qwen2.5-VL, InternVL3, LLaVA, Gemini-2.5-Pro, Llama 4
Also in ML frameworks and libraries
See all 206| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | ultralyticsPython library and CLI for training and running YOLO vision models | 94 out of 100 |
| 3 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |