#5151st of 53 in Gateways and observability
simple-evals
Reference implementations of zero-shot LLM benchmarks such as MMLU, GPQA and SimpleQA
- Stars
- 4.7k
- License
- MIT
- Last commit
- Apr 2026
- Language
- Python
4,408 on 22 Apr 20264,702 on 11 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
OpenAI is hiring: 829 open roles
Overview
A small Python library from OpenAI that runs language-model evals (MMLU, MATH, GPQA, DROP, MGSM, HumanEval, SimpleQA, BrowseComp, HealthBench) with zero-shot chain-of-thought prompts. Samplers exist for the OpenAI and Anthropic APIs, driven by a CLI. Since July 2025 it is no longer updated for new models and only keeps the HealthBench, BrowseComp and SimpleQA reference implementations.
Who it is for: Engineers reproducing OpenAI benchmark numbers or reading reference eval code
Strengths
- Covers nine benchmarks behind one CLI: --model and --examples flags
- Zero-shot chain-of-thought prompts with simple instructions, documented in the README
- Official reference code for SimpleQA, BrowseComp and HealthBench
- Published results table gives baseline numbers for many OpenAI and third-party models
Weaknesses
- Deprecated: no updates for new models or benchmark results after July 2025
- Maintainers do not monitor issues or PRs, and no new evals are accepted
- No unified setup; each eval and sampler needs its own pip install
- Samplers only for OpenAI and Claude APIs; no local model backend
What it needs
- no GPU
- Needs openai, anthropic, human-eval
- Models: OpenAI API, Anthropic Claude API
Also in Gateways and observability
See all 53| Rank | Project | Score |
|---|---|---|
| 1 | OmniRouteLLM gateway routing one OpenAI-compatible endpoint across 372 providers | 96 out of 100 |
| 2 | headroomCompresses tool outputs, logs and history before they reach the LLM | 92 out of 100 |
| 3 | new-apiGateway that unifies LLM providers behind OpenAI, Anthropic and Gemini APIs | 91 out of 100 |
| #4 | MLflowTracing, evaluation and prompt registry for LLM apps, plus ML tracking | 91 out of 100 |
| #5 | LiteLLMOpenAI-format gateway and Python SDK for calling 100+ LLM providers | 88 out of 100 |