Skip to content

Benchmark

SWE-bench Multimodal

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
12
Scores recorded
24
Sources
3
IQ weight
1.07

Score distribution

46.8%
Mean
54.1%
Median
12.7%
Std dev
30.7%
10th percentile
61.4%
90th percentile
28.1%
Min
28.1%74.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.07 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Fable 5.2· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim74.1%
2Claude Opus 5.5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim61.4%
3Claude Opus 5.5Opus 5.5 System Card capability section· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.61.4%
4Claude Opus 5Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59.4%
5Claude Opus 5Opus 5.5 System Card capability section· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59.4%
6Claude Opus 5System Card capability summary · adaptive thinking · max effort · default sampling · 5 trials · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59.4%
7Claude Fable 5.1Opus 5.5 System Card capability section· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.7%
8Claude Fable 5.1System Card capability summary · adaptive thinking · max effort · default sampling · 5 trials · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.7%
9Claude Sonnet 5.5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim54.3%
10Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.3%
11Claude Fable 5Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.1%
12Claude Fable 5System Card capability summary · adaptive thinking · max effort · default sampling · 5 trials · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.1%
13Inkling 2 Small· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim43.1%
14Qwen 3.8 27B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim38.6%
15Qwen 3.8 27BQwen3.8-27B model card · Claude Code · public dev split · Opus 4.7 System Card Appendix 8.3 modifications· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.38.6%
16Claude Opus 4.8· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim38.4%
17Claude Opus 4.8Opus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.38.4%
18Claude Opus 4.8Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.38.4%
19Claude Opus 4.7internal harness · adaptive thinking · max effort · avg@5· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.34.5%
20Claude Opus 4.7Opus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.34.5%
21Claude Haiku 5.5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim30.7%
22Claude Haiku 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.30.7%
23Claude Sonnet 5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim28.1%
24Claude Sonnet 5internal harness · adaptive thinking · max effort · 5 trials· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.28.1%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links. · Data by LLM Stats · Hugging Face official benchmark leaderboards; licence varies by benchmark