Introducing SWE-bench Verified

As part of our Preparedness Framework, OpenAI develops a range of metrics to track, evaluate, and forecast models’ abilities to act autonomously. The ability to autonomously complete software engineering tasks is a key component of our Medium risk level in the Model Autonomy risk category. Evaluating these capabilities is challenging due to the complexity of software engineering tasks, the difficulty of accurately assessing generated code, and the challenge of simulating real-world development scenarios. Therefore, our approach to Preparedness must also involve careful examination of evaluations themselves, to reduce the potential for underestimating or overestimating performance in important risk categories.

One of the most popular evaluation suites for software engineering is SWE-bench(opens in a new window)¹—a benchmark for evaluating large language models’ (LLMs’) abilities to solve real-world software issues sourced from GitHub. The benchmark involves giving agents a code repository and issue description, and challenging them to generate a patch that resolves the problem described by the issue. Coding agents have made impressive progress on SWE-bench, with top scoring agents scoring 20% on SWE-bench and 43% on SWE-bench Lite according to the SWE-bench leaderboard(opens in a new window) as of August 5, 2024.

Our testing identified some SWE-bench tasks which may be hard or impossible to solve, leading to SWE-bench systematically underestimating models’ autonomous software engineering capabilities. We’ve collaborated with the authors of SWE-bench to address those issues in a new release of the benchmark that should provide more accurate evaluations.

Each sample in the SWE-bench test set is created from a resolved GitHub issue in one of 12 open-source Python repositories on GitHub. Each sample has an associated pull request (PR), which includes both the solution code and unit tests to verify code correctness. These unit tests fail before the solution code in the PR is added, but pass afterwards, and are therefore called FAIL_TO_PASS tests. Each sample also has associated PASS_TO_PASS tests, which pass both before and after the PR is merged, and are used to check that existing unrelated functionality in the codebase has not been broken by the PR. 

For each sample in SWE-bench, agents are provided with the original text from the GitHub issue, known as the problem statement, and are given access to the codebase. Given these, agents must edit the files in the codebase to resolve the issue. The tests are not shown to the agent.

A proposed edit is evaluated by running both the FAIL_TO_PASS and PASS_TO_PASS tests. If the FAIL_TO_PASS tests pass, this means the edit solves the issue. If the PASS_TO_PASS tests pass, then the edit has not inadvertently broken unrelated sections of the codebase. Both sets of tests are required to pass for the edit to fully resolve the original GitHub issue.

Given the potential relevance of SWE-bench for the Preparedness Framework, we aimed to find ways in which we could improve the robustness and reliability of the benchmark. We identified three major areas for improvement²: 

1. The unit tests used to evaluate the correctness of a solution are often overly specific, and in some cases are even unrelated to the issue. This potentially causes correct solutions to be rejected. 

2. Many samples have an issue description that is underspecified, leading to ambiguity on what the problem is and how it should be solved.

3. It is sometimes difficult to reliably set up the SWE-bench development environments for the agents, inadvertently causing unit tests to fail regardless of the solution. In such cases, perfectly valid solutions might be graded as incorrect.

Here is an example illustrating the first of these issues.

SWE-bench sample scikit-learn__scikit-learn-14520 tasks an agent with solving an issue in the scikit-learn repository(opens in a new window). This problem statement reports that a function’s copy argument could be specified by a user, but is ignored by the library (the behavior is instead hardcoded inside the function):

An agent approaching the above issue would first have to deal with the ambiguity in whether the function’s behavior is intended or a bug, then make changes to the codebase to resolve the issue. Per the SWE-bench setup, any solution the agent proposes then needs to pass the following test, extracted from the PR that originally resolved the issue(opens in a new window):

This test explicitly checks that the solution must raise a DeprecationWarning whenever the copy parameter is used, although the original problem statement in the issue text above does not convey this requirement. Furthermore, even if the agent realized that a DeprecationWarning should be raised, the test also requires the agent to exactly match the deprecation message, which was only arrived at after some discussion in the PR which the agent has no access to.

Note that the agent is only given the problem description from the main issue text, and does not have visibility into the tests that it needs to pass. Given this setup, it would be nearly impossible for an agent to solve this sample in SWE-bench.

To address these issues, we launched a human annotation campaign with professional software developers to screen each sample of the SWE-bench test set for appropriately scoped unit tests and well-specified issue descriptions.

Together with the authors of SWE-bench, we are releasing SWE-bench Verified: a subset of the original test set from SWE-bench, consisting of 500 samples verified to be non-problematic by our human annotators. This version supersedes the original SWE-bench and SWE-bench Lite test sets. Additionally, we are releasing our human annotations for all SWE-bench test samples.

We also collaborated with the SWE-bench authors to develop a new evaluation harness for SWE-bench(opens in a new window) which uses containerized Docker environments to make evaluating on SWE-bench easier and more reliable.

On SWE-bench Verified, GPT-4o resolves 33.2% of samples³, with the best performing open-source scaffold, Agentless, doubling its previous score of 16% on SWE-bench.

We worked with 93 software developers experienced in Python to manually screen SWE-bench samples for quality. We annotated 1,699 random samples from the SWE-bench test set to produce SWE-bench Verified. The following analysis is based on the 1,699 samples.

We annotate samples to capture:

• Whether we consider the issue description to be underspecified and hence unfair to be testing on.

• Whether the FAIL_TO_PASS unit tests filter out valid solutions.

Each annotation criterion has a label ranging [0, 1, 2, 3] in increasing severity. Labels 0 and 1 are minor; labels 2 and 3 are severe and indicate that the sample is inadequate in some way and should be discarded. We choose to annotate across four ordinal categories rather than a single binary label of severe/not severe to capture more granular detail.

Additionally, we rate the difficulty of each sample by having annotators estimate how long it would take for a developer to decide upon and implement the solution, assuming the sample is non-problematic. Finally, we provide a freeform input option to flag any other major issues with the sample (for example, if the FAIL_TO_PASS unit tests are easily gamed, this could lead to an invalid solution being marked as correct).

Our team of engineers first hand-labeled 50 samples to a high degree of confidence for use in annotator onboarding tests. To take part in the annotation campaign, each prospective annotator had to pass our onboarding tests. We provided detailed feedback to each annotator throughout onboarding to better train them for the task.  Annotators were not necessarily prior experts in the codebases relevant to SWE-bench, but were given time to familiarize themselves with each codebase they worked with.

To ensure a high-quality dataset, each sample is labeled 3 times by separate annotators. It is easy to accidentally miss potential issues, and issues themselves can be ambiguous, so we conservatively ensemble annotations by taking the highest-severity label amongst the 3 annotators.

The full text of our annotation rubric can be found here(opens in a new window).

To construct SWE-bench Verified, we filter out any sample from the original test set where either the problem statement or the FAIL_TO_PASS unit tests have an ensemble label of 2 or above in severity. We also filter out all samples that have other major issues flagged. Given our ensembling method, this is equivalent to filtering out samples where any single annotator of three has flagged an issue with the sample. This approach leads to a higher false-positive rate in removing samples, but helps increase our confidence in sample quality for the final dataset. 

We include as many samples with difficulty 1-4 hours and >4 hours as possible, and then we randomly sample the remainder to arrive at the 500 samples that constitute SWE-bench Verified.

The results of our annotations are below:

We see that 38.3% of samples were flagged for underspecified problem statements, and 61.1% were flagged for unit tests that may unfairly mark valid solutions as incorrect. Overall, our annotation process resulted in 68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues. As discussed previously, this filtering process is likely to be overzealous but allows us to have high confidence in the feasibility of the unfiltered samples.

We present a few examples of samples and their annotations below, cherry-picked to illustrate the diversity in sample quality:

The chart below compares the difficulty distributions of the original SWE-bench datasets and our new SWE-bench Verified dataset. We estimate the difficulty distribution of SWE-bench based on our random subset of 1699 samples. Note that while these results provide estimates of the effort necessary to implement a solution (refer to our annotation instructions for the precise wording), they assume a software engineer who is able to figure out the solution. In practice, we expect the baseline solve rate of a typical human software engineer to be lower than 100%.  

We observe that most (77.8%) of the samples in the original SWE-bench dataset were estimated to take less than an hour for an experienced software engineer to complete. Both SWE-bench Lite and our new SWE-bench Verified dataset skews this further, leaving fewer than 10% of issues estimated to take longer than an hour. However, the mechanism underlying this shift is importantly different: SWE-bench Lite subsampled the original dataset to make the benchmark easier, whereas SWE-bench Verified attempts to remove infeasible samples from the dataset. We further explore this effect in the next section.

With our new SWE-bench Verified dataset, we tested GPT-4o’s performance using several open-source scaffolds that performed well on the original SWE-bench leaderboards⁴.

We found that GPT-4o’s performance on the best-performing scaffold reaches 33.2% on SWE-bench Verified, more than doubling its score of 16% on the original SWE-bench. In general, this validates our initial suspicion that the original SWE-bench dataset underestimates agent abilities. Note that the jump from SWE-bench Lite to SWE-bench Verified is not as significant, because SWE-bench Lite was already filtered in a way that makes it easier(opens in a new window) than the full dataset, though that process would not fully capture the same issues as our filtering procedure.

The increase in performance when evaluating on SWE-bench Verified may partly be explained by shifting the distribution toward easier samples (as shown in earlier analyses). However, our goal is not to inflate benchmark scores, but to make sure that the benchmark faithfully represents model capability at any given difficulty level.

We investigate this by plotting performance stratified by difficulty. If our new dataset merely shifted the difficulty distribution to contain more easy samples, the stratified performance within each category would not change, as appears to be the case going from the original SWE-bench to SWE-bench Lite. We instead observe that performance increases within individual difficulty categories when moving to SWE-bench Verified, which is consistent with the intended effect of removing impossible samples from all categories instead of removing difficult samples. The effect is clearest in the easiest two buckets of difficulty, where we have the most samples.

We use SWE-bench as one of several evaluations tracking the Medium risk level of the Model Autonomy risk category in our Preparedness Framework. Tracking catastrophic risk levels via evaluations relies on ensuring that we can trust evaluation results and are calibrated about what the scores entail.

Our experiences suggest that we should:

Invest in deeply understanding our benchmarks. Although SWE-bench was designed thoughtfully, it underestimates model capabilities due to the issues mentioned in this blogpost. As our systems get closer to AGI, we need to evaluate them on increasingly more challenging tasks. This also elevates the level of expertise and care needed to curate and verify benchmarks to ensure that they are sufficiently challenging and robust (a case where work like CriticGPT, which explores ways in which AI can assist with annotation pipelines, may be helpful).

Account for progress in the ecosystem. Community-led progress in agent scaffolding highlights the need to consider potential external enhancements to a model when assessing risk. Looking at the difference between the worst- and best-performing scaffolds for a given model on the SWE-bench leaderboards(opens in a new window), we can see that, for example, GPT-4’s performance on SWE-bench Lite varies between 2.7% using an early RAG-based scaffold and 28.3% using CodeR. Thus the Preparedness Framework calls for evaluations to be run continually and as often as needed to identify any non-trivial capability change; which includes before, during, and even after training, where models can be enhanced via integration with external systems. Furthermore, curating evaluations is an ecosystem-wide effort, and we hope to continue collaborating with researchers on building trustworthy, high-quality evaluations.

Be cognizant of limitations. Evaluations based on static datasets are inherently limited, and SWE-bench is no exception. Given that the benchmark is composed of scrapes of public GitHub repos, large foundation models that are pre-trained on internet text are likely to be contaminated on the tasks. Furthermore, SWE-bench only covers a narrow distribution of the Medium risk level for model autonomy and so must be supplemented with other evaluations. 

We believe in an empirical and scientific approach to tracking and protecting against catastrophic risk. Building and continually improving evaluations is a key element of this work. There remains much to be done, and we’re eager to see more work from the community in contributing valuable benchmarks like SWE-bench. 

SWE-bench Verified is available for download here(opens in a new window); the full set of our annotations is here(opens in a new window), and our annotation rubric is here(opens in a new window).