Skip to content

Benchmark

SWE-rebench 2025-10

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
30
Scores recorded
31
Sources
1
IQ weight
1.72

Score distribution

27.1%
Mean
26.3%
Median
9.6%
Std dev
16.9%
10th percentile
40.8%
90th percentile
9.8%
Min
9.8%45.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.72 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Opus 4.5tools· SWE-rebench45.5%
2Claude Sonnet 4.5tools· SWE-rebench44.3%
3Gemini 3 Pro Previewtools· SWE-rebench41.6%
4GPT-5 Codextools· SWE-rebench40.8%
5Claude Opus 4.1tools· SWE-rebench38.8%
6GPT 5tools, high· SWE-rebench38.8%
7Claude Sonnet 4tools· SWE-rebench37.3%
8GLM 4.5tools· SWE-rebench34.5%
9MiniMax M2tools· SWE-rebench32.5%
10Qwen3 Codertools· SWE-rebench31.4%
11GLM 4.6tools· SWE-rebench30.2%
12Kimi K2 Instructtools· SWE-rebench29.8%
13Grok Code Fast 1tools· SWE-rebench29.8%
14OpenAI o3tools· SWE-rebench28.2%
15GLM 4.5 Airtools· SWE-rebench26.7%
16DeepSeek V3.1tools· SWE-rebench26.3%
17GPT 5tools, minimal· SWE-rebench26%
18Gemini 2.5 Protools· SWE-rebench23.9%
19GPT OSS 120Btools· SWE-rebench23.5%
20GPT 4.1tools· SWE-rebench22.8%
21Qwen3 235B A22B Instruct 2507tools· SWE-rebench22.7%
22Qwen 3 235B A22B Thinking 2507tools· SWE-rebench21.6%
23o4 Minitools· SWE-rebench20%
24Gemini 2.5 Flashtools· SWE-rebench19.2%
25Qwen3 Next 80B A3B Instructtools· SWE-rebench17.6%
26Qwen3 Coder 30B A3B Instructtools· SWE-rebench17.3%
27DeepSeek R1tools· SWE-rebench17%
28DeepSeek V3tools· SWE-rebench16.9%
29GPT-4.1 minitools· SWE-rebench15.4%
30Qwen3 30B A3Btools, 2507, thinking· SWE-rebench10.6%
31Qwen3 30B A3B Instructtools· SWE-rebench9.8%

Tokens per task, input and output together, as measured by the source. Comparable within this benchmark only.