Skip to content

Benchmark

SWE-Effi Resolved Rate

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
3
Scores recorded
15
Sources
1
IQ weight
No weight published

Score distribution

23.2%
Mean
26%
Median
11.5%
Std dev
10%
10th percentile
38%
90th percentile
4%
Min
4%48%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#Model
1Qwen3 32Bagentless· SWE-Effi data docs48%
2Qwen3 32Bauto-code-rover· SWE-Effi data docs38%
3Qwen3 32Bopenhands· SWE-Effi data docs34%
4Llama 3.3 70B Instructagentless· SWE-Effi data docs28%
5Llama 3.3 70B Instructauto-code-rover· SWE-Effi data docs28%
6Qwen3 32Bagentless-mini· SWE-Effi data docs28%
7Qwen3 32Bswe-agent· SWE-Effi data docs28%
8GPT-4o miniagentless· SWE-Effi data docs26%
9GPT-4o miniagentless-mini· SWE-Effi data docs20%
10Llama 3.3 70B Instructopenhands· SWE-Effi data docs20%
11GPT-4o miniauto-code-rover· SWE-Effi data docs12%
12Llama 3.3 70B Instructswe-agent· SWE-Effi data docs12%
13GPT-4o miniopenhands· SWE-Effi data docs11.9%
14GPT-4o miniswe-agent· SWE-Effi data docs10%
15Llama 3.3 70B Instructagentless-mini· SWE-Effi data docs4%

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: SWE-Effi (Centre for Software Excellence), Apache-2.0; resolved rate converted to percent; resource figures are per-task means (licence)