Skip to content

Benchmark

hal-swe-bench-verified-mini

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
12
Scores recorded
18
Sources
1
IQ weight
1.61

Score distribution

44.5%
Mean
50%
Median
16.5%
Std dev
24%
10th percentile
61%
90th percentile
6%
Min
6%72%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.61 for chat models, measured over 12 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Sonnet 4.5high· HAL (Princeton)72%
2Claude Sonnet 4.5· HAL (Princeton)68%
3Claude Opus 4.1· HAL (Princeton)61%
4Claude Opus 4.1high· HAL (Princeton)54%
5Claude Sonnet 3.7high· HAL (Princeton)54%
6o4 Minilow· HAL (Princeton)54%
7Claude Opus 4· HAL (Princeton)50%
8Claude Sonnet 3.7· HAL (Princeton)50%
9o4 Minihigh· HAL (Princeton)50%
10GPT 5medium· HAL (Princeton)46%
11OpenAI o3medium· HAL (Princeton)46%
12Claude Haiku 4.5high· HAL (Princeton)44%
13GPT 4.1· HAL (Princeton)44%
14Claude Opus 4high· HAL (Princeton)30%
15Claude Haiku 4.5· HAL (Princeton)24%
16DeepSeek V3· HAL (Princeton)24%
17Gemini 2.0 Flash· HAL (Princeton)24%
18DeepSeek R1· HAL (Princeton)6%

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: HAL (Princeton) — cite the HAL paper