#66th of 16 in Observability
deepeval
Pytest-style framework for unit testing LLM apps, RAG pipelines and agents
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 18.7k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
DeepEval is a Python framework that runs LLM evaluations as pytest-style tests via `deepeval test run`. It ships metrics such as G-Eval, DAG, task completion, tool correctness, faithfulness and hallucination, scored by an LLM judge of your choice or local NLP models. It supports end-to-end, trace-based and component-level evals, synthetic dataset generation, and standard benchmarks like MMLU and GSM8K.
Who it is for: Engineers testing LLM apps, RAG pipelines and agents in CI
Strengths
- Metrics for RAG, agents, multi-turn chats, MCP and multimodal outputs
- Pytest-style tests run in CI with `deepeval test run`
- Tracing via @observe supports component-level and agent trajectory evals
- Integrations for LangChain, LangGraph, CrewAI, LlamaIndex, OpenAI Agents, Pydantic AI and others
Weaknesses
- LLM-judge metrics need a model; the quickstart uses an OpenAI API key
- Report sharing and production monitoring sit in the hosted Confident AI platform
- README pushes account login; test cases are logged to the cloud once logged in
- Judge-based scores depend on the chosen judge model and vary between runs
What it needs
- GPU optional
- Needs OpenAI API key or custom LLM judge, Confident AI account (optional)
- Models: OpenAI, Anthropic, custom LLMs, local NLP models
Also in Observability
See all 16| Rank | Project | Score |
|---|---|---|
| 1 | LangfuseTracing, prompt management and evals for LLM apps on ClickHouse | 75 out of 100 |
| 2 | MLflowTracing, evals, prompt registry and AI gateway plus classic ML tracking | 72 out of 100 |
| 3 | promptfooCLI for evaluating and red-teaming prompts, agents and RAG | 72 out of 100 |
| #4 | PhoenixLLM tracing, evaluation and prompt management built on OpenTelemetry | 71 out of 100 |
| #5 | OpikTrace, evaluate and monitor LLM apps and agents, Apache-2.0 end to end | 66 out of 100 |