Skip to content

Benchmark

Multi-SWE-bench Flash All Verified

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
6
Scores recorded
6
Sources
1
IQ weight
0.20

Score distribution

19.9%
Mean
21.3%
Median
3.7%
Std dev
15.7%
10th percentile
25%
90th percentile
15.3%
Min
15.3%25%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.20 for chat models, measured over 6 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 4MopenHands, 20250712_MopenHands_Claude-4-Sonnet· Multi-SWE-bench Experiments data docs25%
2OpenAI o3MopenHands, 20250712_MopenHands_OpenAI-o3· Multi-SWE-bench Experiments data docs23.7%
3Claude Sonnet 3.7MopenHands, 20250712_MopenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs21.3%
4Kimi K2MopenHands, 20250718_MopenHands_Kimi-K2· Multi-SWE-bench Experiments data docs18.3%
5Gemini 2.5 ProMopenHands, 20250712_MopenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs15.7%
6DeepSeek R1MopenHands, 20250712_MopenHands_DeepSeek-R1-0528· Multi-SWE-bench Experiments data docs15.3%

Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)