AgentBench

AgentBench is a benchmark designed to answer a question the AI industry keeps bumping into: when you hand a large language model real tools and a real goal, how well does it actually perform? A model that aces a multiple-choice quiz can still fumble a multi-step task that requires reading a screen, choosing the right tool, and recovering when something breaks. AgentBench was built to measure exactly that gap.

The benchmark was introduced in the paper “AgentBench: Evaluating LLMs as Agents” (arXiv:2308.03688) by Xiao Liu, Hao Yu, and colleagues at the THUDM lab at Tsinghua University. It was published as a poster at ICLR 2024 and released alongside a public GitHub repository containing all code, environments, and an integrated evaluation package.

Why a New Benchmark Was Needed

By 2023, the AI community had dozens of benchmarks for testing an LLM‘s ability to answer questions, translate text, or write code. But AI agents don’t just answer questions — they interact with environments over multiple turns, use tools, handle unexpected outcomes, and plan sequences of actions that may span dozens of steps. Existing benchmarks were measuring the wrong thing: the quality of a single response, not the quality of a whole decision-making process.

Think of it this way: a driving test that only checks whether you can name the traffic signs doesn’t tell anyone whether you can actually merge onto a highway. AgentBench was designed to be the highway.

The Eight Environments

What makes AgentBench distinctive is its breadth. Rather than testing agents in a single domain, it measures general-purpose capability across eight environments, each requiring different skills:

• Operating System (OS): Genuine bash interactions inside an Ubuntu Docker container — file management, system commands, multi-step operational tasks. Metric: success rate.
• Database (DB): Authentic SQL interfaces with varied query types, testing whether an agent can navigate real databases end to end. Metric: success rate.
• Knowledge Graph (KG): Partially observable graph reasoning — the agent must plan, search, and use tools to answer questions from incomplete information. Metric: answer F1.
• Digital Card Game (DCG): A turn-based strategy game (Aquawar from the 2021 THU Agent Competition) that tests tactical decision-making. Metric: win rate.
• Lateral Thinking Puzzles (LTP): An agent must deduce a mystery by asking yes/no/irrelevant questions — a test of creative, non-linear reasoning.
• Household (ALFWorld): Embodied text-based chores in a virtual home — “put a pan on the dining table” — testing spatial planning and instruction following. Metric: success rate.
• Web Shopping (WebShop): A simulated e-commerce site where the agent must find and purchase specific items through pure prompting, with no task-specific training. Metric: success rate.
• Web Browsing (Mind2Web): Multi-website navigation involving click, select, and type actions across diverse domains. Metric: success rate.

Five of these environments were built from scratch; three were recompiled from existing datasets. Each is packaged as a Docker image so every model runs in an isolated, reproducible setting.

What the Benchmark Found

The original study evaluated 29 models, including commercial API-based systems (GPT-4, Claude, ChatGPT) and open-source models (LLaMA-2, CodeLlama, Vicuna, ChatGLM). The headline result was stark: commercial models scored an average of 2.32 on the benchmark’s overall scale, while open-source models averaged 0.51 — more than a fourfold gap.

The benchmark also tracked why agents failed, categorizing each outcome: success, task limit exceeded (TLE — the agent ran out of reasoning turns), context limit exceeded (CLE), invalid format, or invalid action. These failure categories turned out to be the real insight. The most common cause of breakdown was not a lack of knowledge but poor long-term reasoning, decision-making, and instruction following. Models that could chat fluently in a single turn often lost the thread when tasks required sustained planning across many interactions.

A finding that surprised the research community: training on code did not uniformly improve agent performance. It helped in some environments and hurt in others — an ambivalent result that challenged the popular assumption that “code training = better agents.”

From Text Agents to Visual Agents

The AgentBench team extended their framework with VisualAgentBench (VAB) (arXiv:2408.06327, published at ICLR 2025), which evaluates large multimodal models — AI that can process images and video — as visual agents across five environments: VAB-OmniGibson (embodied household tasks), VAB-Minecraft (open-world building), VAB-Mobile (mobile GUI interaction), VAB-WebArena-Lite (realistic web navigation), and VAB-CSS (visual design). VisualAgentBench also includes trajectory training data for behavior cloning, beginning to bridge the gap between evaluation and improvement.

Where AgentBench Fits in the Landscape

AgentBench is one of several foundational agent evaluation and benchmarking frameworks. GAIA tests general AI assistant capability through practical multi-step tasks. SWE-bench zeros in on software engineering by challenging agents to resolve real GitHub issues. WebArena tests web interaction in realistic website environments. What sets AgentBench apart is its deliberately broad scope — eight environments across code, reasoning, games, domestic tasks, and the web — giving it a claim to measuring general-purpose agent capability rather than performance in a single vertical.

Sources

• Xiao Liu et al., “AgentBench: Evaluating LLMs as Agents” (arXiv 2308.03688, August 7, 2023; published at ICLR 2024). THUDM, Tsinghua University.
• THUDM/AgentBench — Official GitHub repository (code, environments, and evaluation package)