Skip to content

Benchmark

SWE-Lancer offline accuracy

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
2
Scores recorded
2
Sources
1
IQ weight
No weight published

Score distribution

18.3%
Mean
28.4%
Median
10.2%
Std dev
8.1%
10th percentile
28.4%
90th percentile
8.1%
Min
8.1%28.4%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#ModelScore
1o1offline-198· SWE-Lancer offline leaderboard28.4%
2GPT 4ooffline-198· SWE-Lancer offline leaderboard8.1%

Data credits: SWE-Lancer community leaderboard. Offline task subset as sized in the publisher README; aggregate accuracy and earned payouts.