Benchmark
Open Agent Leaderboard SWE-bench
Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.
- Models measured
- 5
- Scores recorded
- 25
- Sources
- 1
- IQ weight
- 0.19
Score distribution
- 63.6%
- Mean
- 60.6%
- Median
- 8%
- Std dev
- 54.5%
- 10th percentile
- 74.2%
- 90th percentile
- 52%
- Min
This benchmark has a weight of 0.19 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Claude Opus 4.5agent: openai_solo, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs | 80.7% |
| 2 | Gemini 3 Pro Previewagent: smolagents_code, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs | 75.8% |
| 3 | Claude Opus 4.5agent: claude_code, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs | 74.2% |
| 4 | DeepSeek V3.2agent: openai_solo, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 73.7% |
| 5 | Gemini 3 Pro Previewagent: openai_solo, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs | 72.3% |
| 6 | Gemini 3 Pro Previewagent: tool_calling_with_shortlisting, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs | 71% |
| 7 | Gemini 3 Pro Previewagent: tool_calling, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs | 71% |
| 8 | DeepSeek V3.2agent: tool_calling_with_shortlisting, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 68.8% |
| 9 | DeepSeek V3.2agent: tool_calling, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 68.8% |
| 10 | Gemini 3 Pro Previewagent: claude_code, task: swebench, route: openai/gcp· Open Agent Leaderboard results data docs | 67% |
| 11 | Claude Opus 4.5agent: smolagents_code, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs | 65% |
| 12 | DeepSeek V3.2agent: claude_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 64% |
| 13 | Claude Opus 4.5agent: tool_calling_with_shortlisting, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs | 60.6% |
| 14 | Claude Opus 4.5agent: tool_calling, task: swebench, route: openai/aws· Open Agent Leaderboard results data docs | 60.6% |
| 15 | GPT 5.2agent: claude_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 58% |
| 16 | Kimi K2.5agent: smolagents_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 57.6% |
| 17 | Kimi K2.5agent: tool_calling_with_shortlisting, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 57.1% |
| 18 | Kimi K2.5agent: tool_calling, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 57.1% |
| 19 | GPT 5.2agent: tool_calling_with_shortlisting, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 57% |
| 20 | GPT 5.2agent: tool_calling, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 57% |
| 21 | Kimi K2.5agent: openai_solo, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 56.7% |
| 22 | DeepSeek V3.2agent: smolagents_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 56% |
| 23 | GPT 5.2agent: openai_solo, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 54.5% |
| 24 | GPT 5.2agent: smolagents_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 52.5% |
| 25 | Kimi K2.5agent: claude_code, task: swebench, route: openai/azure· Open Agent Leaderboard results data docs | 52% |
Data credits: Open Agent Leaderboard results (Exgentic), CDLA-Permissive-2.0; agreement text at https://cdla.dev/permissive-2-0/. Fractions shown as percentages. (licence)