Skip to content

Benchmark

SWE-bench-Live Full

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
3
Scores recorded
4
Sources
1
IQ weight
No weight published

Score distribution

16%
Mean
18.6%
Median
4.1%
Std dev
9.1%
10th percentile
19.3%
90th percentile
9.1%
Min
9.1%19.3%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#ModelScore
1Claude Sonnet 3.7scaffold: openhands, set: full, instances: 1319, submitted: 2025-05-01· SWE-bench-Live leaderboard data19.3%
2GPT 4.1scaffold: swe-agent, set: full, instances: 1319, submitted: 2025-05-01· SWE-bench-Live leaderboard data18.6%
3Claude Sonnet 3.7scaffold: swe-agent, set: full, instances: 1319, submitted: 2025-05-01· SWE-bench-Live leaderboard data17.1%
4GPT 4oscaffold: swe-agent, set: full, instances: 1319, submitted: 2025-05-01· SWE-bench-Live leaderboard data9.1%

Data credits: SWE-bench-Live (Microsoft and contributors), swe-bench-live.github.io. Resolve rates computed from the published report counts.