#3535th of 57 in Gateways and observability
evalscope
Evaluation and stress-testing framework for LLM, VLM and embedding models
- Stars
- 3.7k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- Python
2,709 on 25 Apr 20263,710 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
EvalScope is a Python CLI and library from the ModelScope community that runs benchmarks (MMLU, C-Eval, GSM8K, SWE-bench and many others) against OpenAI-compatible or Anthropic APIs. It also load-tests model services, reporting TTFT and TPOT, and shows results in a web dashboard with model comparison. It can run agent benchmarks in Docker or E2B sandboxes and wraps OpenCompass, VLMEvalKit and RAGEval as backends.
Who it is for: Engineers benchmarking LLMs and load-testing model serving endpoints
Strengths
- Large built-in benchmark set covering text, multimodal, embedding, reranker, code and agent tasks
- Evaluates any OpenAI-compatible endpoint with one command, no local GPU required
- Includes perf stress testing with TTFT/TPOT and multi-turn trace replay
- Web dashboard, arena pairwise mode, and extension points for custom datasets and metrics
Weaknesses
- Requires Python 3.11 or newer
- Agent and code benchmarks need Docker or an E2B sandbox
- Fast-moving feature set with past breaking changes, such as the v1.0 refactor
- No Docker image or compose file provided in the repository
What it needs
- GPU optional
- Needs Docker or E2B (agent/code benchmarks), OpenCompass, VLMEvalKit, RAGEval (optional backends)
- Models: OpenAI-compatible API, Anthropic API, LLM, VLM, Embedding
What changed
- v1.15.0 ↗ (opens in a new tab)Breaking
“Python 3.11 or newer is required. Sandbox extras use ms-enclave[docker,e2b]>=0.0.9. E2B supports only the documented benchmarks; external agents and Harbor/Pier execution are outside this integration.”
- v1.14.1 ↗ (opens in a new tab)Breaking
“Install litellm explicitly when using that backend. Core OpenAI, System One and related paths no longer install it transitively; third-party backends may still depend on it.”
- v1.14.0 ↗ (opens in a new tab)Breaking
“Evaluation semantics changed for several benchmarks; check evaluationversion. MMBench v1.1 changes input prompts and requires new predictions. Scoring-only fixes can use rerunreview=True / --rerun-re…”
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in Gateways and observability
See all 57| Rank | Project | Score |
|---|---|---|
| 1 | OmniRouteLLM gateway routing one OpenAI-compatible endpoint across 372 providers | 96 out of 100 |
| 2 | headroomCompresses tool outputs, logs and history before they reach the LLM | 92 out of 100 |
| 3 | new-apiGateway that unifies LLM providers behind OpenAI, Anthropic and Gemini APIs | 91 out of 100 |
| #4 | MLflowTracing, evaluation and prompt registry for LLM apps, plus ML tracking | 91 out of 100 |
| #5 | LiteLLMOpenAI-format gateway and Python SDK for calling 100+ LLM providers | 88 out of 100 |