#153153rd of 153 in Agent frameworks and SDKs
agentbench
Benchmark suite for evaluating LLMs as agents across multi-turn environments
- Stars
- 3.8k
- License
- Apache-2.0
- Last commit
- Feb 2026
- Language
- Python
3,340 on 25 Apr 20263,828 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
AgentBench evaluates LLMs acting as agents in environments such as OS, database, knowledge graph, house-holding and web shopping. The current branch is a function-calling version integrated with AgentRL, with Docker Compose deployment for five tasks (alfworld, dbbench, knowledgegraph, os_interaction, webshop). Older versions (v0.1, v0.2) cover eight environments and run via a Python controller, task workers and an assigner.
Who it is for: Researchers benchmarking LLM agents on multi-turn tasks
Strengths
- Covers 8 environments in the original version, including OS, DB, KG and web shopping
- Docker Compose file starts controller, task workers, Redis and Freebase server together
- Task workers are scalable per task and expose a controller-based task assignment flow
- Extension guide documents how to add new tasks
Weaknesses
- Webshop needs about 16 GB RAM per worker; alfworld leaks memory and disk until restart
- Knowledge graph task needs a separately downloaded Freebase dataset or an unstable online SPARQL service
- Original version pins old dependencies such as numpy 1.23 and Python 3.9
- Function-calling version supports only five of the original eight tasks
What it needs
- RAM ≥ 16 GB
- no GPU
- Needs Docker, Docker Compose, Redis 7+, MySQL 8, Freebase/Virtuoso data, Python 3.9
- Models: OpenAI API (gpt-3.5-turbo-0613 in quick start), function-calling LLMs
- port 5000
Also in Agent frameworks and SDKs
See all 153| Rank | Project | Score |
|---|---|---|
| 1 | ppt-masterAgent skill that turns documents into natively editable PowerPoint decks | 96 out of 100 |
| 2 | copilotkitReact and Angular SDK for adding agent chat and generative UI to apps | 92 out of 100 |
| 3 | openai-agents-pythonPython SDK for multi-agent workflows with handoffs, guardrails and tracing | 92 out of 100 |
| #4 | mastraTypeScript framework for building AI agents, workflows and MCP servers | 92 out of 100 |
| #5 | browser-usePython library that lets an LLM agent control a web browser | 90 out of 100 |