Skip to content

Benchmark

Multi-SWE-bench Python Verified

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
10
Scores recorded
30
Sources
1
IQ weight
1.25

Score distribution

29.3%
Mean
28.6%
Median
15.1%
Std dev
4.4%
10th percentile
46.4%
90th percentile
2%
Min
2%52.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.25 for chat models, measured over 10 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 3.7MopenHands, 20250329_OpenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs52.2%
2Gemini 2.5 ProMagentLess, 20250426_Agentless_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs49%
3o1MagentLess, 20250329_Agentless_OpenAI-o1· Multi-SWE-bench Experiments data docs48.2%
4o3 MiniMagentLess, 20250329_Agentless_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs46.4%
5Claude Sonnet 3.7MSWE-agent, 20250329_SWE-agent_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs45.8%
6Gemini 2.5 ProMopenHands, 20250426_OpenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs45.8%
7Claude Sonnet 3.7MagentLess, 20250329_Agentless_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs44.6%
8Claude 3.5 SonnetMagentLess, 20250329_Agentless_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs42.4%
9DeepSeek R1MagentLess, 20250329_Agentless_DeepSeek-R1· Multi-SWE-bench Experiments data docs42.2%
10DeepSeek V3MagentLess, 20250329_Agentless_DeepSeek-V3· Multi-SWE-bench Experiments data docs41%
11Claude 3.5 SonnetMopenHands, 20250329_OpenHands_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs39%
12Llama 4 MaverickMagentLess, 20250426_Agentless_Llama-4-Maverick· Multi-SWE-bench Experiments data docs37.8%
13GPT 4oMagentLess, 20250329_Agentless_GPT-4o-1120· Multi-SWE-bench Experiments data docs36.2%
14o1MSWE-agent, 20250329_SWE-agent_OpenAI-o1· Multi-SWE-bench Experiments data docs28.8%
15o3 MiniMSWE-agent, 20250329_SWE-agent_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs28.6%
16DeepSeek V3MopenHands, 20250329_OpenHands_DeepSeek-V3· Multi-SWE-bench Experiments data docs27.8%
17Gemini 2.5 ProMSWE-agent, 20250426_SWE-agent_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs27.8%
18Qwen2.5 72B InstructMagentLess, 20250329_Agentless_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs26.8%
19DeepSeek R1MopenHands, 20250329_OpenHands_DeepSeek-R1· Multi-SWE-bench Experiments data docs26%
20GPT 4oMopenHands, 20250329_OpenHands_GPT-4o-1120· Multi-SWE-bench Experiments data docs25.6%
21Claude 3.5 SonnetMSWE-agent, 20250329_SWE-agent_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs24.8%
22o3 MiniMopenHands, 20250329_OpenHands_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs20.4%
23GPT 4oMSWE-agent, 20250329_SWE-agent_GPT-4o-1120· Multi-SWE-bench Experiments data docs18.8%
24o1MopenHands, 20250329_OpenHands_OpenAI-o1· Multi-SWE-bench Experiments data docs16%
25Llama 4 MaverickMopenHands, 20250426_OpenHands_Llama-4-Maverick· Multi-SWE-bench Experiments data docs14.4%
26Qwen2.5 72B InstructMSWE-agent, 20250329_SWE-agent_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs8.6%
27Qwen2.5 72B InstructMopenHands, 20250329_OpenHands_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs4.4%
28DeepSeek V3MSWE-agent, 20250329_SWE-agent_DeepSeek-V3· Multi-SWE-bench Experiments data docs4.2%
29DeepSeek R1MSWE-agent, 20250329_SWE-agent_DeepSeek-R1· Multi-SWE-bench Experiments data docs2%
30Llama 4 MaverickMSWE-agent, 20250426_SWE-agent_Llama-4-Maverick· Multi-SWE-bench Experiments data docs2%

Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)