Skip to content

Benchmark

SWE-rebench 2026-03

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
18
Scores recorded
18
Sources
1
IQ weight
2.15

Score distribution

46.8%
Mean
51.3%
Median
10.8%
Std dev
36.1%
10th percentile
54.3%
90th percentile
13.5%
Min
13.5%63%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.15 for chat models, measured over 18 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1GPT 5.5tools, xhigh· SWE-rebench63%
2Claude Opus 4.8tools, xhigh· SWE-rebench56.5%
3Claude Sonnet 4.6tools· SWE-rebench54.3%
4Gemini 3.1 Pro Previewtools· SWE-rebench53.9%
5Claude Opus 4.7tools, high· SWE-rebench53.5%
6Gemini 3.5 Flashtools· SWE-rebench53.5%
7GLM 5.2tools, high· SWE-rebench53.5%
8GLM 5.1tools· SWE-rebench53%
9Claude Opus 4.6tools, high· SWE-rebench51.3%
10Kimi K2.6tools· SWE-rebench47.4%
11DeepSeek V4 Protools, high· SWE-rebench46.7%
12MiniMax M3tools· SWE-rebench46.1%
13MiMo V2.5 Protools· SWE-rebench43.9%
14GLM 4.7tools· SWE-rebench41.7%
15Qwen3.6 27Btools· SWE-rebench38.7%
16DeepSeek V4 Flashtools, high· SWE-rebench36.1%
17Qwen3.6 35B A3Btools· SWE-rebench36.1%
18Gemma 4 31B ITtools· SWE-rebench13.5%

Tokens per task, input and output together, as measured by the source. Comparable within this benchmark only.