#153153rd of 153 in Agent frameworks and SDKs

agentbench

Benchmark suite for evaluating LLMs as agents across multi-turn environments

Stars
3.8k
License
Apache-2.0
Last commit
Feb 2026
Language
Python
Stars, last 29 weeks+488
25 Apr 2026: 3,340 stars2 May 2026: 3,363 stars9 May 2026: 3,381 stars16 May 2026: 3,405 stars23 May 2026: 3,424 stars30 May 2026: 3,440 stars6 Jun 2026: 3,453 stars13 Jun 2026: 3,473 stars20 Jun 2026: 3,489 stars27 Jun 2026: 3,508 stars4 Jul 2026: 3,531 stars11 Jul 2026: 3,544 stars18 Jul 2026: 3,566 stars25 Jul 2026: 3,589 stars1 Aug 2026: 3,616 stars8 Aug 2026: 3,641 stars15 Aug 2026: 3,657 stars22 Aug 2026: 3,667 stars29 Aug 2026: 3,689 stars5 Sept 2026: 3,703 stars12 Sept 2026: 3,716 stars19 Sept 2026: 3,733 stars26 Sept 2026: 3,745 stars3 Oct 2026: 3,757 stars10 Oct 2026: 3,764 stars17 Oct 2026: 3,777 stars24 Oct 2026: 3,793 stars31 Oct 2026: 3,807 stars7 Nov 2026: 3,818 stars14 Nov 2026: 3,828 stars

3,340 on 25 Apr 20263,828 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

AgentBench evaluates LLMs acting as agents in environments such as OS, database, knowledge graph, house-holding and web shopping. The current branch is a function-calling version integrated with AgentRL, with Docker Compose deployment for five tasks (alfworld, dbbench, knowledgegraph, os_interaction, webshop). Older versions (v0.1, v0.2) cover eight environments and run via a Python controller, task workers and an assigner.

Who it is for: Researchers benchmarking LLM agents on multi-turn tasks

Strengths

  • Covers 8 environments in the original version, including OS, DB, KG and web shopping
  • Docker Compose file starts controller, task workers, Redis and Freebase server together
  • Task workers are scalable per task and expose a controller-based task assignment flow
  • Extension guide documents how to add new tasks

Weaknesses

  • Webshop needs about 16 GB RAM per worker; alfworld leaks memory and disk until restart
  • Knowledge graph task needs a separately downloaded Freebase dataset or an unstable online SPARQL service
  • Original version pins old dependencies such as numpy 1.23 and Python 3.9
  • Function-calling version supports only five of the original eight tasks

What it needs

  • RAM ≥ 16 GB
  • no GPU
  • Needs Docker, Docker Compose, Redis 7+, MySQL 8, Freebase/Virtuoso data, Python 3.9
  • Models: OpenAI API (gpt-3.5-turbo-0613 in quick start), function-calling LLMs
  • port 5000

Also in Agent frameworks and SDKs

See all 153
Also in Agent frameworks and SDKs
RankProjectScore
1ppt-masterAgent skill that turns documents into natively editable PowerPoint decks Live demo ↗ (opens in a new tab)78k starsMIT96 out of 100
2copilotkitReact and Angular SDK for adding agent chat and generative UI to apps38.8k starsMIT92 out of 100
3openai-agents-pythonPython SDK for multi-agent workflows with handoffs, guardrails and tracing30.7k starsMIT92 out of 100
#4mastraTypeScript framework for building AI agents, workflows and MCP servers29.6k starscustom license92 out of 100
#5browser-usePython library that lets an LLM agent control a web browser Live demo ↗ (opens in a new tab)121.4k starsMIT90 out of 100