Skip to content

Benchmark

SWE-rebench 2025-11

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
30
Scores recorded
31
Sources
1
IQ weight
1.79

Score distribution

37.9%
Mean
39.6%
Median
15.5%
Std dev
17.9%
10th percentile
56.6%
90th percentile
6.8%
Min
6.8%60.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.79 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Sonnet 4.5tools· SWE-rebench60.9%
2Claude Opus 4.5tools· SWE-rebench60.4%
3GPT-5.1 Codex Maxtools· SWE-rebench58.3%
4Gemini 3 Pro Previewtools· SWE-rebench56.6%
5GPT 5tools, high· SWE-rebench55.7%
6Claude Sonnet 4tools· SWE-rebench53.6%
7Claude Opus 4.1tools· SWE-rebench51.9%
8OpenAI o3tools· SWE-rebench51.9%
9GPT-5 Codextools· SWE-rebench51.1%
10DeepSeek V3.2tools· SWE-rebench46.4%
11GLM 4.6tools· SWE-rebench46%
12GLM 4.5tools· SWE-rebench44.3%
13Devstral 2 123b Instruct 2512tools· SWE-rebench43.8%
14MiniMax M2.1tools· SWE-rebench43.4%
15Kimi K2 Instructtools· SWE-rebench41.7%
16MiniMax M2tools· SWE-rebench39.6%
17o4 Minitools· SWE-rebench39.6%
18Qwen3 Codertools· SWE-rebench39.1%
19Grok Code Fast 1tools· SWE-rebench36.2%
20GPT 5tools, minimal· SWE-rebench34.9%
21GLM 4.5 Airtools· SWE-rebench33.6%
22GPT OSS 120Btools· SWE-rebench30.6%
23Qwen3 235B A22B Instruct 2507tools· SWE-rebench26.5%
24DeepSeek R1tools· SWE-rebench23.9%
25Qwen3 Coder 30B A3B Instructtools· SWE-rebench23%
26GPT 4.1tools· SWE-rebench20%
27Qwen 3 235B A22B Thinking 2507tools· SWE-rebench17.9%
28Qwen3 Next 80B A3B Instructtools· SWE-rebench17.9%
29GPT-4.1 minitools· SWE-rebench12.3%
30Qwen3 30B A3Btools, 2507, thinking· SWE-rebench8.5%
31Qwen3 30B A3B Instructtools· SWE-rebench6.8%

Tokens per task, input and output together, as measured by the source. Comparable within this benchmark only.