Skip to content

Benchmark

Multi-SWE-bench Java Verified

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
11
Scores recorded
31
Sources
1
IQ weight
1.52

Score distribution

14.1%
Mean
12.5%
Median
8.1%
Std dev
5.5%
10th percentile
22.7%
90th percentile
2.3%
Min
2.3%39.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.52 for chat models, measured over 11 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5.2InfCode, 20260131_InfCode_GPT-5.2· Multi-SWE-bench Experiments data docs39.1%
2Gemini 2.5 ProMSWE-agent, 20250426_MSWE-agent_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs28.9%
3Claude Sonnet 3.7MSWE-agent, 20250329_MSWE-agent_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs23.4%
4DeepSeek R1MagentLess, 20250329_MagentLess_DeepSeek-R1· Multi-SWE-bench Experiments data docs22.7%
5Claude Sonnet 3.7MopenHands, 20250329_MopenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs21.9%
6Gemini 2.5 ProMagentLess, 20250426_MagentLess_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs21.9%
7o1MSWE-agent, 20250329_MSWE-agent_OpenAI-o1· Multi-SWE-bench Experiments data docs21.9%
8o1MagentLess, 20250329_MagentLess_OpenAI-o1· Multi-SWE-bench Experiments data docs21.1%
9Claude 3.5 SonnetMSWE-agent, 20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs20.3%
10o3 MiniMSWE-agent, 20250329_MSWE-agent_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs16.4%
11Claude 3.5 SonnetMagentLess, 20250329_MagentLess_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs14.8%
12Claude 3.5 SonnetMopenHands, 20250329_MopenHands_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs14.8%
13Llama 4 MaverickMagentLess, 20250426_MagentLess_Llama-4-Maverick· Multi-SWE-bench Experiments data docs14.8%
14Claude Sonnet 3.7MagentLess, 20250329_MagentLess_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs14.1%
15Gemini 2.5 ProMopenHands, 20250426_MopenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs12.5%
16GPT 4oMSWE-agent, 20250329_MSWE-agent_GPT-4o-1120· Multi-SWE-bench Experiments data docs12.5%
17DeepSeek V3MSWE-agent, 20250329_MSWE-agent_DeepSeek-V3· Multi-SWE-bench Experiments data docs11.7%
18GPT 4oMagentLess, 20250329_MagentLess_GPT-4o-1120· Multi-SWE-bench Experiments data docs11.7%
19Qwen2.5 72B InstructMagentLess, 20250329_MagentLess_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs10.9%
20o3 MiniMopenHands, 20250329_MopenHands_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs10.2%
21DeepSeek R1MSWE-agent, 20250329_MSWE-agent_DeepSeek-R1· Multi-SWE-bench Experiments data docs9.4%
22DeepSeek V3MopenHands, 20250329_MopenHands_DeepSeek-V3· Multi-SWE-bench Experiments data docs9.4%
23GPT 4oMopenHands, 20250329_MopenHands_GPT-4o-1120· Multi-SWE-bench Experiments data docs9.4%
24DeepSeek R1MopenHands, 20250329_MopenHands_DeepSeek-R1· Multi-SWE-bench Experiments data docs8.6%
25DeepSeek V3MagentLess, 20250329_MagentLess_DeepSeek-V3· Multi-SWE-bench Experiments data docs7%
26Llama 4 MaverickMopenHands, 20250426_MopenHands_Llama-4-Maverick· Multi-SWE-bench Experiments data docs6.3%
27Llama 4 MaverickMSWE-agent, 20250426_MSWE-agent_Llama-4-Maverick· Multi-SWE-bench Experiments data docs6.3%
28o3 MiniMagentLess, 20250329_MagentLess_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs5.5%
29o1MopenHands, 20250329_MopenHands_OpenAI-o1· Multi-SWE-bench Experiments data docs3.9%
30Qwen2.5 72B InstructMopenHands, 20250329_MopenHands_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs3.1%
31Qwen2.5 72B InstructMSWE-agent, 20250329_MSWE-agent_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs2.3%

Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)