SWE-bench

SWE-bench is the standard benchmark for evaluating large language models and autonomous coding agents on real-world software engineering tasks. Given a complete codebase and a natural-language issue (a bug report or feature request sourced from GitHub), an agent must generate a code patch that resolves the issue and passes the repository’s automated test suite. Introduced by Princeton University researchers at ICLR 2024, SWE-bench has become the primary measure used to rank AI coding capabilities.

A Worked Example

Imagine a popular open-source Python library has a bug: it crashes when processing a specific file format. A developer opens a GitHub issue describing the crash, including the steps to reproduce it and the expected behavior. In a SWE-bench task, an AI agent is given the entire codebase at the moment before the bug was fixed, along with the text of that issue. The agent must read the issue, navigate the relevant files, understand the problem, write the necessary code changes, and submit a patch. If the agent’s patch fixes the bug and causes the library’s automated tests to pass, the agent succeeds.

How Evaluation Works: The Fail-to-Pass Metric

The heart of SWE-bench is its Fail-to-Pass evaluation method. Because the benchmark is built from real GitHub pull requests (PRs), each task comes with a set of tests that are known to fail before the fix and pass after it. This gives the benchmark an objective, reproducible signal: did the agent’s patch actually fix the problem, or did it just happen to pass unrelated tests?

Evaluation runs inside Docker containers for reproducibility. The agent’s patch is applied to the repository’s base commit, the test suite is executed, and the result is scored on whether the specific Fail-to-Pass tests now pass.

Dataset Variants

SWE-bench is available in several versions to suit different evaluation needs:

• SWE-bench Full: 2,294 task instances sourced from 12 popular Python repositories—the complete benchmark.
• SWE-bench Lite: A curated subset of 300 instances, designed for faster iteration during development.
• SWE-bench Verified: 500 instances that have been reviewed and confirmed as solvable by professional software engineers. This version addresses concerns that some original tasks may have been ambiguous or unsolvable.
• SWE-bench Multimodal: Extends the benchmark to visual software domains (introduced at ICLR 2025).
• SWE-bench Multilingual: Extends coverage beyond Python to other programming languages.

Why SWE-bench Matters

As agentic AI systems become more capable, the question shifts from “can a model write code?” to “can a model fix real bugs in real codebases?” SWE-bench answers the second question. It tests an agent’s ability to navigate unfamiliar code, reason about complex systems, and produce correct patches—skills that matter far more than generating isolated code snippets.

This makes SWE-bench a critical tool for agent evaluation and benchmarking. Leading AI labs use SWE-bench scores to rank their models’ software engineering capabilities, and the leaderboard at swebench.com is the primary public tracker for progress in AI coding agents.

Key Takeaways

• SWE-bench evaluates an AI agent’s ability to solve real GitHub issues by generating code patches—not just write isolated code snippets.
• Success is measured by Fail-to-Pass tests: specific tests that fail before the fix and must pass after the agent’s patch is applied.
• The benchmark is available in multiple sizes (Full, Lite, Verified) and domains (Multimodal, Multilingual).
• SWE-bench is the industry standard for ranking the coding capabilities of AI models and agents.