Skip to content

Benchmark

SWE-bench-Live Windows

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
5
Scores recorded
5
Sources
1
IQ weight
0.21

Score distribution

22.9%
Mean
21.7%
Median
4.9%
Std dev
16%
10th percentile
30%
90th percentile
16%
Min
16%30%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.21 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 4.5scaffold: win-agent, set: windows, instances: 50, submitted: 2026-01-08· SWE-bench-Live leaderboard data30%
2GPT 5.5scaffold: win-agent, set: windows, instances: 60, submitted: 2026-06-06, effort: medium· SWE-bench-Live leaderboard data26.7%
3DeepSeek V4 Proscaffold: win-agent, set: windows, instances: 60, submitted: 2026-06-06· SWE-bench-Live leaderboard data21.7%
4DeepSeek V3.1 Terminusscaffold: win-agent, set: windows, instances: 50, submitted: 2026-01-08· SWE-bench-Live leaderboard data20%
5Gemini 3 Flashscaffold: win-agent, set: windows, instances: 50, submitted: 2026-01-08· SWE-bench-Live leaderboard data16%

Data credits: SWE-bench-Live (Microsoft and contributors), swe-bench-live.github.io. Resolve rates computed from the published report counts.