Skip to content

Benchmark

Deepswe Pass@1

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
26
Scores recorded
69
Sources
1
IQ weight
2.17

Score distribution

54.9%
Mean
58.1%
Median
16.7%
Std dev
30.5%
10th percentile
72.8%
90th percentile
1.5%
Min
1.5%74.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.17 for chat models, measured over 26 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

Showing 51–69 of 69 results.

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: Epoch AI Benchmarking Hub, CC BY 4.0 (licence)