Skip to content

Benchmark

Open Agent Leaderboard SWE-bench

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
5
Scores recorded
25
Sources
1
IQ weight
0.19

Score distribution

63.6%
Mean
60.6%
Median
8%
Std dev
54.5%
10th percentile
74.2%
90th percentile
52%
Min
52%80.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.19 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 4.5agent: openai_solo, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs80.7%
2Gemini 3 Pro Previewagent: smolagents_code, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs75.8%
3Claude Opus 4.5agent: claude_code, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs74.2%
4DeepSeek V3.2agent: openai_solo, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs73.7%
5Gemini 3 Pro Previewagent: openai_solo, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs72.3%
6Gemini 3 Pro Previewagent: tool_calling_with_shortlisting, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs71%
7Gemini 3 Pro Previewagent: tool_calling, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs71%
8DeepSeek V3.2agent: tool_calling_with_shortlisting, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs68.8%
9DeepSeek V3.2agent: tool_calling, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs68.8%
10Gemini 3 Pro Previewagent: claude_code, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs67%
11Claude Opus 4.5agent: smolagents_code, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs65%
12DeepSeek V3.2agent: claude_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs64%
13Claude Opus 4.5agent: tool_calling_with_shortlisting, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs60.6%
14Claude Opus 4.5agent: tool_calling, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs60.6%
15GPT 5.2agent: claude_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs58%
16Kimi K2.5agent: smolagents_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs57.6%
17Kimi K2.5agent: tool_calling_with_shortlisting, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs57.1%
18Kimi K2.5agent: tool_calling, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs57.1%
19GPT 5.2agent: tool_calling_with_shortlisting, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs57%
20GPT 5.2agent: tool_calling, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs57%
21Kimi K2.5agent: openai_solo, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs56.7%
22DeepSeek V3.2agent: smolagents_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs56%
23GPT 5.2agent: openai_solo, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs54.5%
24GPT 5.2agent: smolagents_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs52.5%
25Kimi K2.5agent: claude_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs52%

Data credits: Open Agent Leaderboard results (Exgentic), CDLA-Permissive-2.0; agreement text at https://cdla.dev/permissive-2-0/. Fractions shown as percentages. (licence)