Skip to content

Benchmark

SWE-rebench 2025-08

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
32
Scores recorded
33
Sources
1
IQ weight
2.13

Score distribution

28.1%
Mean
29.2%
Median
12.4%
Std dev
8.1%
10th percentile
42.3%
90th percentile
2.3%
Min
2.3%49.4%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.13 for chat models, measured over 32 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Sonnet 4tools· SWE-rebench49.4%
2GPT 5tools, high· SWE-rebench46.5%
3GLM 4.5tools· SWE-rebench45%
4Kimi K2 Instructtools· SWE-rebench42.3%
5Grok 4tools· SWE-rebench41.7%
6Qwen3 Codertools· SWE-rebench40.7%
7GPT 4.1tools· SWE-rebench37.4%
8Grok Code Fast 1tools· SWE-rebench37.3%
9OpenAI o3tools· SWE-rebench36.5%
10Qwen3 235B A22B Instruct 2507tools· SWE-rebench35.8%
11GLM 4.5 Airtools· SWE-rebench34.7%
12Kimi K2tools· SWE-rebench34.6%
13GPT 5tools, minimal· SWE-rebench33.5%
14DeepSeek V3.1tools· SWE-rebench32.7%
15Claude 3.5 Sonnettools· SWE-rebench31.6%
16Qwen3 Next 80B A3B Instructtools· SWE-rebench29.7%
17o4 Minitools· SWE-rebench29.2%
18DeepSeek R1tools· SWE-rebench29.2%
19Qwen3 Coder 30B A3B Instructtools· SWE-rebench29.2%
20Gemini 2.5 Protools· SWE-rebench28.8%
21GPT OSS 120Btools· SWE-rebench26.5%
22Qwen 3 235B A22B Thinking 2507tools· SWE-rebench26.5%
23Gemini 2.5 Flashtools· SWE-rebench25.8%
24DeepSeek V3tools· SWE-rebench24.6%
25GPT-4.1 minitools· SWE-rebench19.2%
26Qwen 3 235B A22Btools· SWE-rebench18.5%
27Gemini 2.0 Flashtools· SWE-rebench15.5%
28Llama 3.3 70B Instructtext· SWE-rebench10%
29Qwen2.5 72B Instructtext· SWE-rebench9.6%
30GPT OSS 20btools· SWE-rebench8.1%
31Qwen3 32Btools· SWE-rebench8.1%
32Llama 4 Mavericktext· SWE-rebench6.6%
33Gemma 3 27B ITtext· SWE-rebench2.3%

Tokens per task, input and output together, as measured by the source. Comparable within this benchmark only.