Skip to content

Benchmark

SWE-Marathon v1.1

Scores are percentages. Higher is better.

Models and systems measured
7
Scores recorded
7
Sources
1
IQ weight
0.39

Score distribution

39.7%
Mean
42.5%
Median
7.5%
Std dev
31.9%
10th percentile
46%
90th percentile
30%
Min
30%50%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.39 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Claude Opus 5native provider harnesses · 8 trials/task · resolution rate · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.50%
2Grok 4.7native provider harnesses · 8 trials/task · resolution rate · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.46%
3GPT-5.6 Solnative provider harnesses · 8 trials/task · resolution rate · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.5%
4GPT-5.6 Terranative provider harnesses · 8 trials/task · resolution rate · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.32.5%
5Claude Sonnet 5native provider harnesses · 8 trials/task · resolution rate · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.30%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Fable 5 with Opus 4.8 fallbacknative provider harnesses · 8 trials/task · resolution rate · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.45%
Grok 4.6native provider harnesses · 8 trials/task · resolution rate · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.31.9%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.