Skip to content

Benchmark

hal-core-bench-hard

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages.

Models measured
17
Scores recorded
24
Sources
1
IQ weight
1.36

Score distribution

31.9%
Mean
33.3%
Median
17.2%
Std dev
11.1%
10th percentile
51.1%
90th percentile
8.9%
Min
8.9%77.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.36 for chat models, measured over 17 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Opus 4.5· HAL (Princeton)77.8%
2Claude Sonnet 4.5· HAL (Princeton)62.2%
3Claude Opus 4.1· HAL (Princeton)51.1%
4Claude Sonnet 4· HAL (Princeton)46.7%
5Claude Sonnet 4.5high· HAL (Princeton)44.4%
6Claude Opus 4.1high· HAL (Princeton)42.2%
7Claude Opus 4.5high· HAL (Princeton)42.2%
8Gemini 3 Pro Previewhigh· HAL (Princeton)40%
9Claude Sonnet 3.7high· HAL (Princeton)37.8%
10Claude Sonnet 3.7· HAL (Princeton)35.6%
11o4 Minihigh· HAL (Princeton)35.6%
12Claude Sonnet 4high· HAL (Princeton)33.3%
13GPT 4.1· HAL (Princeton)33.3%
14GPT 5medium· HAL (Princeton)26.7%
15OpenAI o3medium· HAL (Princeton)24.4%
16Gemini 2.5 Pro Preview 06-05· HAL (Princeton)22.2%
17DeepSeek V3.1· HAL (Princeton)20%
18DeepSeek V3· HAL (Princeton)17.8%
19o4 Minilow· HAL (Princeton)17.8%
20Claude Haiku 4.5· HAL (Princeton)11.1%
21Gemini 2.0 Flash· HAL (Princeton)11.1%
22GPT OSS 120B· HAL (Princeton)11.1%
23GPT OSS 120Bhigh· HAL (Princeton)11.1%
24DeepSeek R1· HAL (Princeton)8.9%

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: HAL (Princeton) — cite the HAL paper