#6767th of 105 in ML frameworks and libraries
opencompass
Evaluation harness for benchmarking LLMs and VLMs on 100+ datasets
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 7.5k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Aug 2026
- Language
- Python
6,775 on 1 Apr 20267,533 on 21 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
OpenCompass runs benchmark evaluations for language models across knowledge, reasoning, coding, science, long-context and safety datasets. It covers dataset and model configuration, inference, scoring and result summaries, using HuggingFace, LMDeploy or vLLM backends and API models from OpenAI, Anthropic, Gemini and others. Scoring can be rule-based or LLM-as-judge, and VLMEvalKit is integrated for vision-language sets.
Who it is for: Engineers and researchers benchmarking LLMs and VLMs reproducibly
Strengths
- Over 100 datasets plus VLMEvalKit integration in one config-driven workflow
- Supports HuggingFace, LMDeploy and vLLM backends and hosted API models
- Task partitioning, concurrent and distributed execution for large evaluation runs
- LLM-as-judge, math verification and cascaded evaluators for custom scoring
Weaknesses
- Inference backends have conflicting dependencies; separate virtual environments recommended
- Pyext-based code datasets (APPS, TACO, LiveCodeBench) need Python 3.10, not 3.12
- No Docker or compose files provided in the repository
- README documents no web UI; evaluations are configured and run from Python files
What it needs
- GPU optional
- Needs conda, HuggingFace, LMDeploy, vLLM
- Models: HuggingFace, LMDeploy, vLLM, OpenAI, Anthropic
Also in ML frameworks and libraries
See all 105| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 92 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 91 out of 100 |