Skip to content

Benchmark

Multi-SWE-bench Lite All Verified

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
10
Scores recorded
30
Sources
1
IQ weight
1.48

Score distribution

8.3%
Mean
7.5%
Median
4%
Std dev
4.3%
10th percentile
12.3%
90th percentile
1%
Min
1%18.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.48 for chat models, measured over 10 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 3.7MopenHands, 20250329_MopenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs18.3%
2Gemini 2.5 ProMopenHands, 20250426_MopenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs18%
3Claude Sonnet 3.7MSWE-agent, 20250329_MSWE-agent_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs15.3%
4Claude 3.5 SonnetMopenHands, 20250329_MopenHands_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs12.3%
5Gemini 2.5 ProMagentLess, 20250426_MagentLess_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs12.3%
6o1MagentLess, 20250329_MagentLess_OpenAI-o1· Multi-SWE-bench Experiments data docs10.8%
7Llama 4 MaverickMagentLess, 20250426_MagentLess_Llama-4-Maverick· Multi-SWE-bench Experiments data docs10.3%
8DeepSeek R1MagentLess, 20250329_MagentLess_DeepSeek-R1· Multi-SWE-bench Experiments data docs10%
9Gemini 2.5 ProMSWE-agent, 20250426_MSWE-agent_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs9.8%
10Claude 3.5 SonnetMSWE-agent, 20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs9.3%
11Claude Sonnet 3.7MagentLess, 20250329_MagentLess_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs9.3%
12o3 MiniMagentLess, 20250329_MagentLess_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs9%
13Claude 3.5 SonnetMagentLess, 20250329_MagentLess_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs8.3%
14DeepSeek V3MagentLess, 20250329_MagentLess_DeepSeek-V3· Multi-SWE-bench Experiments data docs8%
15DeepSeek V3MopenHands, 20250329_MopenHands_DeepSeek-V3· Multi-SWE-bench Experiments data docs7.5%
16GPT 4oMagentLess, 20250329_MagentLess_GPT-4o-1120· Multi-SWE-bench Experiments data docs7.5%
17o3 MiniMSWE-agent, 20250329_MSWE-agent_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs7.5%
18o1MSWE-agent, 20250329_MSWE-agent_OpenAI-o1· Multi-SWE-bench Experiments data docs7.3%
19DeepSeek R1MopenHands, 20250329_MopenHands_DeepSeek-R1· Multi-SWE-bench Experiments data docs7%
20GPT 4oMopenHands, 20250329_MopenHands_GPT-4o-1120· Multi-SWE-bench Experiments data docs6.8%
21Qwen2.5 72B InstructMagentLess, 20250329_MagentLess_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs6%
22o3 MiniMopenHands, 20250329_MopenHands_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs5.8%
23Llama 4 MaverickMopenHands, 20250426_MopenHands_Llama-4-Maverick· Multi-SWE-bench Experiments data docs5.5%
24GPT 4oMSWE-agent, 20250329_MSWE-agent_GPT-4o-1120· Multi-SWE-bench Experiments data docs5.3%
25DeepSeek V3MSWE-agent, 20250329_MSWE-agent_DeepSeek-V3· Multi-SWE-bench Experiments data docs5%
26o1MopenHands, 20250329_MopenHands_OpenAI-o1· Multi-SWE-bench Experiments data docs4.8%
27Llama 4 MaverickMSWE-agent, 20250426_MSWE-agent_Llama-4-Maverick· Multi-SWE-bench Experiments data docs4.3%
28DeepSeek R1MSWE-agent, 20250329_MSWE-agent_DeepSeek-R1· Multi-SWE-bench Experiments data docs3.8%
29Qwen2.5 72B InstructMopenHands, 20250329_MopenHands_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs2.3%
30Qwen2.5 72B InstructMSWE-agent, 20250329_MSWE-agent_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs1%

Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)