Skip to content

Benchmark

SWE-rebench 2025-12

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
24
Scores recorded
25
Sources
1
IQ weight
1.55

Score distribution

40.9%
Mean
38.4%
Median
14.3%
Std dev
22.7%
10th percentile
60%
90th percentile
11.5%
Min
11.5%63.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.55 for chat models, measured over 24 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Opus 4.5tools· SWE-rebench63.3%
2GPT 5.2tools, xhigh· SWE-rebench61.5%
3Gemini 3 Flashtools· SWE-rebench60%
4Gemini 3 Pro Previewtools· SWE-rebench58.9%
5Claude Sonnet 4.5tools· SWE-rebench57.5%
6GPT-5.1 Codex Maxtools· SWE-rebench56.7%
7GPT-5.1 Codextools· SWE-rebench54.6%
8GLM 4.7tools· SWE-rebench51.3%
9DeepSeek V3.2tools· SWE-rebench48.5%
10GPT-5 minitools, high· SWE-rebench44.3%
11Kimi K2 Thinkingtools· SWE-rebench40.5%
12GLM 4.6tools· SWE-rebench40.1%
13Kimi K2 Instructtools· SWE-rebench38.4%
14Qwen3 Codertools· SWE-rebench38.3%
15MiniMax M2.1tools· SWE-rebench37.3%
16GPT OSS 120Btools, high· SWE-rebench37%
17Devstral 2 123b Instruct 2512tools· SWE-rebench36.6%
18Grok Code Fast 1tools· SWE-rebench35.9%
19GLM 4.5 Airtools· SWE-rebench34.1%
20DeepSeek R1tools· SWE-rebench26%
21Qwen3 235B A22B Instruct 2507tools· SWE-rebench24.6%
22Qwen3 Coder 30B A3B Instructtools· SWE-rebench23%
23GPT OSS 120Btools· SWE-rebench22.7%
24Qwen3 Next 80B A3B Instructtools· SWE-rebench19.9%
25Qwen3 30B A3B Instructtools· SWE-rebench11.5%

Tokens per task, input and output together, as measured by the source. Comparable within this benchmark only.