Skip to content

Benchmark

SWE-Lancer (IC-Diamond subset)

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
6
Scores recorded
6
Sources
1
IQ weight
0.77

Score distribution

48.9%
Mean
74.6%
Median
37.4%
Std dev
12.4%
10th percentile
100%
90th percentile
7.4%
Min
7.4%100%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.77 for chat models, measured over 6 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim100%
2GPT-5.3 Codex· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim81.4%
3GPT 5.2· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim74.6%
4GPT-4.5 Preview· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim17.4%
5GPT 4o· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim12.4%
6o3 Mini· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim7.4%

Data credits: Data by LLM Stats