#1212th of 14 in Observability
evals
Python framework and registry for evaluating LLMs and LLM-based systems
- Stars
- 19.6k
- License
- custom license
- Last commit
- Apr 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
OpenAI is hiring: 804 open roles
Overview
Runs evals against OpenAI models and LLM-built systems, using a registry of existing evals plus custom ones defined as JSON data and YAML config. Model-graded evals need no evaluation code, and a Completion Function Protocol covers prompt chains and tool-using agents. Results can optionally be logged to Snowflake.
Who it is for: Engineers who want repeatable evals of LLM prompts, models and agents
Strengths
- Basic and model-graded evals need only JSON data and YAML, no code
- Completion Function Protocol supports prompt chains and tool-using agents
- Registry of existing evals, with private evals possible on your own data
- Optional result logging to Snowflake
Weaknesses
- Setup is documented around an OpenAI API key; other providers are not described
- Registry data is stored in Git-LFS and must be pulled separately
- Not accepting contributed evals with custom code
- Known issue: runs can hang after the final report until interrupted
What it needs
- no GPU
- Needs OpenAI API key, Git-LFS, Snowflake (optional)
- Models: OpenAI models, custom completion functions
Also in Observability
See all 14| Rank | Project | Score |
|---|---|---|
| 1 | LangfuseTracing, prompt management and evals for LLM apps on ClickHouse | 74 out of 100 |
| 2 | PhoenixLLM tracing, evaluation and prompt management built on OpenTelemetry | 72 out of 100 |
| 3 | MLflowTracing, evals, prompt registry and AI gateway plus classic ML tracking | 70 out of 100 |
| #4 | promptfooCLI for evaluating and red-teaming prompts, agents and RAG | 70 out of 100 |
| #5 | OpikTrace, evaluate and monitor LLM apps and agents, Apache-2.0 end to end | 64 out of 100 |