Benchmark
Multi-SWE-bench C++ Verified
Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.
- Models measured
- 11
- Scores recorded
- 31
- Sources
- 1
- IQ weight
- 1.92
Score distribution
- 6.1%
- Mean
- 5.4%
- Median
- 4.9%
- Std dev
- 1.6%
- 10th percentile
- 11.6%
- 90th percentile
- 0%
- Min
This benchmark has a weight of 1.92 for chat models, measured over 11 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | GPT 5InfCode, 20251110_InfCode_GPT-5· Multi-SWE-bench Experiments data docs | 25.6% |
| 2 | Claude Sonnet 3.7MopenHands, 20250329_MopenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs | 14.7% |
| 3 | Claude 3.5 SonnetMopenHands, 20250329_MopenHands_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs | 12.4% |
| 4 | Claude Sonnet 3.7MSWE-agent, 20250329_MSWE-agent_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs | 11.6% |
| 5 | Gemini 2.5 ProMopenHands, 20250426_MopenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs | 9.3% |
| 6 | Gemini 2.5 ProMSWE-agent, 20250426_MSWE-agent_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs | 8.5% |
| 7 | DeepSeek V3MopenHands, 20250329_MopenHands_DeepSeek-V3· Multi-SWE-bench Experiments data docs | 7.8% |
| 8 | DeepSeek V3MSWE-agent, 20250329_MSWE-agent_DeepSeek-V3· Multi-SWE-bench Experiments data docs | 7.8% |
| 9 | Claude 3.5 SonnetMSWE-agent, 20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs | 7% |
| 10 | GPT 4oMagentLess, 20250329_MagentLess_GPT-4o-1120· Multi-SWE-bench Experiments data docs | 7% |
| 11 | Llama 4 MaverickMSWE-agent, 20250426_MSWE-agent_Llama-4-Maverick· Multi-SWE-bench Experiments data docs | 7% |
| 12 | o3 MiniMopenHands, 20250329_MopenHands_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs | 7% |
| 13 | DeepSeek R1MSWE-agent, 20250329_MSWE-agent_DeepSeek-R1· Multi-SWE-bench Experiments data docs | 6.2% |
| 14 | Llama 4 MaverickMagentLess, 20250426_MagentLess_Llama-4-Maverick· Multi-SWE-bench Experiments data docs | 5.4% |
| 15 | o1MagentLess, 20250329_MagentLess_OpenAI-o1· Multi-SWE-bench Experiments data docs | 5.4% |
| 16 | o3 MiniMSWE-agent, 20250329_MSWE-agent_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs | 5.4% |
| 17 | DeepSeek R1MopenHands, 20250329_MopenHands_DeepSeek-R1· Multi-SWE-bench Experiments data docs | 4.7% |
| 18 | Claude 3.5 SonnetMagentLess, 20250329_MagentLess_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs | 3.9% |
| 19 | GPT 4oMopenHands, 20250329_MopenHands_GPT-4o-1120· Multi-SWE-bench Experiments data docs | 3.9% |
| 20 | o1MopenHands, 20250329_MopenHands_OpenAI-o1· Multi-SWE-bench Experiments data docs | 3.9% |
| 21 | o1MSWE-agent, 20250329_MSWE-agent_OpenAI-o1· Multi-SWE-bench Experiments data docs | 3.9% |
| 22 | Claude Sonnet 3.7MagentLess, 20250329_MagentLess_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs | 3.1% |
| 23 | DeepSeek R1MagentLess, 20250329_MagentLess_DeepSeek-R1· Multi-SWE-bench Experiments data docs | 3.1% |
| 24 | Llama 4 MaverickMopenHands, 20250426_MopenHands_Llama-4-Maverick· Multi-SWE-bench Experiments data docs | 3.1% |
| 25 | Gemini 2.5 ProMagentLess, 20250426_MagentLess_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs | 2.3% |
| 26 | GPT 4oMSWE-agent, 20250329_MSWE-agent_GPT-4o-1120· Multi-SWE-bench Experiments data docs | 2.3% |
| 27 | Qwen2.5 72B InstructMopenHands, 20250329_MopenHands_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs | 2.3% |
| 28 | DeepSeek V3MagentLess, 20250329_MagentLess_DeepSeek-V3· Multi-SWE-bench Experiments data docs | 1.6% |
| 29 | o3 MiniMagentLess, 20250329_MagentLess_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs | 1.6% |
| 30 | Qwen2.5 72B InstructMagentLess, 20250329_MagentLess_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs | 0.775% |
| 31 | Qwen2.5 72B InstructMSWE-agent, 20250329_MSWE-agent_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs | 0% |
Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)