#3535th of 57 in Gateways and observability

evalscope

Evaluation and stress-testing framework for LLM, VLM and embedding models

Stars
3.7k
License
Apache-2.0
Last commit
Nov 2026
Last release
Nov 2026
Language
Python
Stars, last 29 weeks+1,001
25 Apr 2026: 2,709 stars2 May 2026: 2,733 stars9 May 2026: 2,759 stars16 May 2026: 2,786 stars23 May 2026: 2,824 stars30 May 2026: 2,859 stars6 Jun 2026: 2,889 stars13 Jun 2026: 2,926 stars20 Jun 2026: 2,961 stars27 Jun 2026: 2,987 stars4 Jul 2026: 3,023 stars11 Jul 2026: 3,058 stars18 Jul 2026: 3,101 stars25 Jul 2026: 3,137 stars1 Aug 2026: 3,172 stars8 Aug 2026: 3,211 stars15 Aug 2026: 3,241 stars22 Aug 2026: 3,287 stars29 Aug 2026: 3,333 stars5 Sept 2026: 3,374 stars12 Sept 2026: 3,411 stars19 Sept 2026: 3,447 stars26 Sept 2026: 3,481 stars3 Oct 2026: 3,508 stars10 Oct 2026: 3,528 stars17 Oct 2026: 3,570 stars24 Oct 2026: 3,598 stars31 Oct 2026: 3,637 stars7 Nov 2026: 3,676 stars14 Nov 2026: 3,710 stars

2,709 on 25 Apr 20263,710 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

EvalScope is a Python CLI and library from the ModelScope community that runs benchmarks (MMLU, C-Eval, GSM8K, SWE-bench and many others) against OpenAI-compatible or Anthropic APIs. It also load-tests model services, reporting TTFT and TPOT, and shows results in a web dashboard with model comparison. It can run agent benchmarks in Docker or E2B sandboxes and wraps OpenCompass, VLMEvalKit and RAGEval as backends.

Who it is for: Engineers benchmarking LLMs and load-testing model serving endpoints

Strengths

  • Large built-in benchmark set covering text, multimodal, embedding, reranker, code and agent tasks
  • Evaluates any OpenAI-compatible endpoint with one command, no local GPU required
  • Includes perf stress testing with TTFT/TPOT and multi-turn trace replay
  • Web dashboard, arena pairwise mode, and extension points for custom datasets and metrics

Weaknesses

  • Requires Python 3.11 or newer
  • Agent and code benchmarks need Docker or an E2B sandbox
  • Fast-moving feature set with past breaking changes, such as the v1.0 refactor
  • No Docker image or compose file provided in the repository

What it needs

  • GPU optional
  • Needs Docker or E2B (agent/code benchmarks), OpenCompass, VLMEvalKit, RAGEval (optional backends)
  • Models: OpenAI-compatible API, Anthropic API, LLM, VLM, Embedding

What changed

  1. “Python 3.11 or newer is required. Sandbox extras use ms-enclave[docker,e2b]>=0.0.9. E2B supports only the documented benchmarks; external agents and Harbor/Pier execution are outside this integration.”

  2. “Install litellm explicitly when using that backend. Core OpenAI, System One and related paths no longer install it transitively; third-party backends may still depend on it.”

  3. “Evaluation semantics changed for several benchmarks; check evaluationversion. MMBench v1.1 changes input prompts and requires new predictions. Scoring-only fixes can use rerunreview=True / --rerun-re…”

From the release notes. No flag means the note does not say so, not that the upgrade is safe.

Also in Gateways and observability

See all 57
Also in Gateways and observability
RankProjectScore
1OmniRouteLLM gateway routing one OpenAI-compatible endpoint across 372 providers88.5k starsMIT96 out of 100
2headroomCompresses tool outputs, logs and history before they reach the LLM79.9k starsApache-2.092 out of 100
3new-apiGateway that unifies LLM providers behind OpenAI, Anthropic and Gemini APIs52.1k starsAGPL-3.0known advisory in latest version91 out of 100
#4MLflowTracing, evaluation and prompt registry for LLM apps, plus ML tracking Live demo ↗ (opens in a new tab)28.9k starsApache-2.091 out of 100
#5LiteLLMOpenAI-format gateway and Python SDK for calling 100+ LLM providers63.5k starscustom license88 out of 100