Skip to content

Benchmark

Multi-SWE-bench JavaScript Verified

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
11
Scores recorded
31
Sources
1
IQ weight
3.27

Score distribution

4.1%
Mean
3.4%
Median
3.8%
Std dev
0.843%
10th percentile
7.6%
90th percentile
0.562%
Min
0.562%17.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 3.27 for chat models, measured over 11 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 4.5RepoRepair, 20260202_RepoRepair_Claude-4.5-Sonnet· Multi-SWE-bench Experiments data docs17.1%
2Gemini 2.5 ProMopenHands, 20250426_MopenHands_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs16.3%
3Gemini 2.5 ProMagentLess, 20250426_MagentLess_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs8.7%
4Gemini 2.5 ProMSWE-agent, 20250426_MSWE-agent_Gemini-2.5-Pro· Multi-SWE-bench Experiments data docs7.6%
5Claude Sonnet 3.7MopenHands, 20250329_MopenHands_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs5.1%
6Llama 4 MaverickMagentLess, 20250426_MagentLess_Llama-4-Maverick· Multi-SWE-bench Experiments data docs5.1%
7o1MagentLess, 20250329_MagentLess_OpenAI-o1· Multi-SWE-bench Experiments data docs5.1%
8Claude Sonnet 3.7MSWE-agent, 20250329_MSWE-agent_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs4.8%
9Llama 4 MaverickMopenHands, 20250426_MopenHands_Llama-4-Maverick· Multi-SWE-bench Experiments data docs4.8%
10DeepSeek R1MagentLess, 20250329_MagentLess_DeepSeek-R1· Multi-SWE-bench Experiments data docs4.5%
11Claude 3.5 SonnetMSWE-agent, 20250329_MSWE-agent_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs4.2%
12o1MSWE-agent, 20250329_MSWE-agent_OpenAI-o1· Multi-SWE-bench Experiments data docs4.2%
13o3 MiniMSWE-agent, 20250329_MSWE-agent_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs4.2%
14o1MopenHands, 20250329_MopenHands_OpenAI-o1· Multi-SWE-bench Experiments data docs3.7%
15DeepSeek V3MagentLess, 20250329_MagentLess_DeepSeek-V3· Multi-SWE-bench Experiments data docs3.4%
16o3 MiniMopenHands, 20250329_MopenHands_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs3.4%
17o3 MiniMagentLess, 20250329_MagentLess_OpenAI-o3-mini-high· Multi-SWE-bench Experiments data docs2.8%
18DeepSeek R1MopenHands, 20250329_MopenHands_DeepSeek-R1· Multi-SWE-bench Experiments data docs2.5%
19DeepSeek V3MSWE-agent, 20250329_MSWE-agent_DeepSeek-V3· Multi-SWE-bench Experiments data docs2.5%
20Llama 4 MaverickMSWE-agent, 20250426_MSWE-agent_Llama-4-Maverick· Multi-SWE-bench Experiments data docs2.2%
21Claude 3.5 SonnetMagentLess, 20250329_MagentLess_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs2%
22Claude 3.5 SonnetMopenHands, 20250329_MopenHands_Claude-3.5-Sonnet(Oct)· Multi-SWE-bench Experiments data docs2%
23Claude Sonnet 3.7MagentLess, 20250329_MagentLess_Claude-3.7-Sonnet· Multi-SWE-bench Experiments data docs2%
24GPT 4oMopenHands, 20250329_MopenHands_GPT-4o-1120· Multi-SWE-bench Experiments data docs2%
25DeepSeek R1MSWE-agent, 20250329_MSWE-agent_DeepSeek-R1· Multi-SWE-bench Experiments data docs1.4%
26GPT 4oMagentLess, 20250329_MagentLess_GPT-4o-1120· Multi-SWE-bench Experiments data docs1.4%
27DeepSeek V3MopenHands, 20250329_MopenHands_DeepSeek-V3· Multi-SWE-bench Experiments data docs1.1%
28GPT 4oMSWE-agent, 20250329_MSWE-agent_GPT-4o-1120· Multi-SWE-bench Experiments data docs0.843%
29Qwen2.5 72B InstructMagentLess, 20250329_MagentLess_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs0.843%
30Qwen2.5 72B InstructMopenHands, 20250329_MopenHands_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs0.843%
31Qwen2.5 72B InstructMSWE-agent, 20250329_MSWE-agent_Qwen2.5-72B-Instruct· Multi-SWE-bench Experiments data docs0.562%

Data credits: Multi-SWE-bench experiments (multi-swe-bench/experiments), Apache-2.0; resolved-rate fractions converted to percentages (licence)