Skip to content

Benchmark

HalluHard Hallucination Rate (coding)

Coding benchmarks compare models on programming tasks. Scores are percentages. Lower is better.

Models measured
30
Scores recorded
39
Sources
1
IQ weight
1.42

Score distribution

42.2%
Mean
40.2%
Median
20.7%
Std dev
16.1%
10th percentile
67.8%
90th percentile
4.3%
Min
4.3%79.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.42 for chat models, measured over 30 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5.4websearch, thinking· HalluHard Leaderboard data docs4.3%
2GPT-6 Astrawebsearch· HalluHard Leaderboard data docs5%
3GPT-6 Astra· HalluHard Leaderboard data docs5.2%
4GPT 5.4thinking· HalluHard Leaderboard data docs11.7%
5Claude Fable 5· HalluHard Leaderboard data docs16.1%
6Claude Fable 5websearch· HalluHard Leaderboard data docs16.1%
7Claude Opus 4.6· HalluHard Leaderboard data docs23.2%
8Claude Opus 4.5· HalluHard Leaderboard data docs25.7%
9Claude Sonnet 5· HalluHard Leaderboard data docs25.8%
10Claude Opus 4.7· HalluHard Leaderboard data docs26.9%
11Gemini 3.1 Pro Preview· HalluHard Leaderboard data docs27%
12Claude Opus 4.5websearch· HalluHard Leaderboard data docs29%
13Claude Sonnet 4.6· HalluHard Leaderboard data docs29.7%
14Gemini 3 Pro Preview· HalluHard Leaderboard data docs31.7%
15GPT 5.2thinking· HalluHard Leaderboard data docs32.2%
16Kimi K2.6thinking· HalluHard Leaderboard data docs35%
17GPT 5.2· HalluHard Leaderboard data docs36.8%
18Claude Sonnet 4.5· HalluHard Leaderboard data docs37.2%
19GPT-5.3websearch· HalluHard Leaderboard data docs39.7%
20GPT-5.3· HalluHard Leaderboard data docs40.2%
21GPT 5thinking· HalluHard Leaderboard data docs41.2%
22Gemini 3 Flash· HalluHard Leaderboard data docs48.3%
23GLM 5thinking· HalluHard Leaderboard data docs49%
24GPT 5· HalluHard Leaderboard data docs50.3%
25GLM 5.2· HalluHard Leaderboard data docs54.5%
26GPT-5 mini· HalluHard Leaderboard data docs54.7%
27Grok 4thinking· HalluHard Leaderboard data docs55.7%
28Kimi K2.5thinking· HalluHard Leaderboard data docs56.2%
29GLM 5websearch, thinking· HalluHard Leaderboard data docs58%
30GLM 4.7thinking· HalluHard Leaderboard data docs59.2%
31DeepSeek V4 Pro· HalluHard Leaderboard data docs61.7%
32Kimi K2 Thinking· HalluHard Leaderboard data docs61.8%
33Claude Haiku 4.5· HalluHard Leaderboard data docs62.5%
34Kimi K2.5websearch· HalluHard Leaderboard data docs63.2%
35DeepSeek Chat· HalluHard Leaderboard data docs67.8%
36GPT-5 nano· HalluHard Leaderboard data docs71%
37DeepSeek Reasoner· HalluHard Leaderboard data docs74.6%
38Grok 4.1 Fastthinking· HalluHard Leaderboard data docs77.3%
39Nemotron 3 Ultra· HalluHard Leaderboard data docs79.3%

Data credits: HalluHard: Fan, Delsad, Flammarion and Andriushchenko (EPFL), arXiv:2602.01031