SWE-bench Verified

Properties

axis: coding
verification: det-test
whatItTests: 500 human-validated real GitHub issues; an agent must produce a patch that passes the repo's existing unit test suite.
discriminates: [build-verify-reflect][debate][ensemble]
saturation: saturated
patternLabFit: Native build-verify-reflect (write patch, run tests, read failures, revise); the test harness is the oracle. Best benchmark for roundsToFirstPass and error-decorrelation (different agents fix different bugs).
source: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)
notes: Saturation moved from mid to saturated (re-verified 2026-10-25): vals.ai now lists SWE-bench Verified as an Archived Benchmark, 'Since performance on this benchmark has saturated, we no longer run this benchmark on new model releases.' Final leaderboard (updated 2026-09-01): Claude Opus 5 (Mini-SWE-agent) first at 97.00%, DeepSeek V4 Pro 0813 at 96.40%, GPT-5.6 Sol at 96.20%. For build-verify-reflect headroom use swe-bench-pro, which was built as its harder successor.

axis: coding
verification: det-test
whatItTests: 500 human-validated real GitHub issues; an agent must produce a patch that passes the repo's existing unit test suite.
discriminates: [build-verify-reflect][debate][ensemble]
saturation: saturated
patternLabFit: Native build-verify-reflect (write patch, run tests, read failures, revise); the test harness is the oracle. Best benchmark for roundsToFirstPass and error-decorrelation (different agents fix different bugs).
source: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)
notes: Saturation moved from mid to saturated (re-verified 2026-10-25): vals.ai now lists SWE-bench Verified as an Archived Benchmark, 'Since performance on this benchmark has saturated, we no longer run this benchmark on new model releases.' Final leaderboard (updated 2026-09-01): Claude Opus 5 (Mini-SWE-agent) first at 97.00%, DeepSeek V4 Pro 0813 at 96.40%, GPT-5.6 Sol at 96.20%. For build-verify-reflect headroom use swe-bench-pro, which was built as its harder successor.