Skip to content

Benchmark

SWE-rebench 2025-09

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
28
Scores recorded
29
Sources
1
IQ weight
1.50

Score distribution

28.4%
Mean
28.6%
Median
8.7%
Std dev
17.6%
10th percentile
40.2%
90th percentile
10.2%
Min
10.2%44.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.50 for chat models, measured over 28 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Sonnet 4.5tools· SWE-rebench44.5%
2GPT-5 Codextools· SWE-rebench41.2%
3Claude Sonnet 4tools· SWE-rebench40.6%
4Claude Opus 4.1tools· SWE-rebench40.2%
5GLM 4.6tools· SWE-rebench37%
6GPT 5tools, high· SWE-rebench36.3%
7OpenAI o3tools· SWE-rebench36.3%
8Qwen3 Codertools· SWE-rebench35.7%
9GLM 4.5tools· SWE-rebench35.1%
10Grok 4tools· SWE-rebench34.6%
11GLM 4.5 Airtools· SWE-rebench31%
12GPT 5tools, minimal· SWE-rebench30.6%
13Grok Code Fast 1tools· SWE-rebench30.1%
14GPT OSS 120Btools· SWE-rebench28.7%
15Qwen3 235B A22B Instruct 2507tools· SWE-rebench28.6%
16GPT 4.1tools· SWE-rebench28.4%
17o4 Minitools· SWE-rebench27.3%
18Kimi K2 Instructtools· SWE-rebench25.9%
19DeepSeek V3.1tools· SWE-rebench24.9%
20Qwen3 Coder 30B A3B Instructtools· SWE-rebench23.3%
21Qwen 3 235B A22B Thinking 2507tools· SWE-rebench22.4%
22DeepSeek V3tools· SWE-rebench22.1%
23Gemini 2.5 Protools· SWE-rebench21.4%
24DeepSeek R1tools· SWE-rebench20%
25Qwen3 Next 80B A3B Instructtools· SWE-rebench19.7%
26Gemini 2.5 Flashtools· SWE-rebench17.6%
27GPT-4.1 minitools· SWE-rebench15.9%
28Qwen3 30B A3Btools, 2507, thinking· SWE-rebench13.1%
29Qwen3 30B A3B Instructtools· SWE-rebench10.2%

Tokens per task, input and output together, as measured by the source. Comparable within this benchmark only.