Benchmark
Multi-SWE-bench Flash All Verified
Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.
- Models measured
- 6
- Scores recorded
- 6
- Sources
- 1
- IQ weight
- 0.20
Score distribution
- 19.9%
- Mean
- 21.3%
- Median
- 3.7%
- Std dev
- 15.7%
- 10th percentile
- 25%
- 90th percentile
- 15.3%
- Min
This benchmark has a weight of 0.20 for chat models, measured over 6 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Claude Sonnet 4MopenHands, 20250712_MopenHands_Claude-4-Sonnet· Multi-SWE-bench Experiments data docs | 25% |
| 2 | OpenAI o3MopenHands, 20250712_MopenHands_OpenAI-o3· Multi-SWE-bench Experiments data docs | 23.7% |
| 3 | Claude Sonnet 3.7MopenHands, 20250712_MopenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs | 21.3% |
| 4 | Kimi K2MopenHands, 20250718_MopenHands_Kimi-K2· Multi-SWE-bench Experiments data docs | 18.3% |
| 5 | Gemini 2.5 ProMopenHands, 20250712_MopenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs | 15.7% |
| 6 | DeepSeek R1MopenHands, 20250712_MopenHands_DeepSeek-R1-0528· Multi-SWE-bench Experiments data docs | 15.3% |
Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)