Skip to content

Benchmark

Confabulation (Hallucination) for RAG (Lech Mazur)

Factuality benchmarks measure how often models get facts right. Scores are percentages. Lower is better.

Models measured
54
Scores recorded
60
Sources
1
IQ weight
2.46

Score distribution

18.8%
Mean
17.1%
Median
6.9%
Std dev
12.3%
10th percentile
28.4%
90th percentile
10.3%
Min
10.3%40.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.46 for chat models, measured over 54 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5medium reasoning· Lech Mazur suite data10.3%
2Gemini 2.5 Pro Preview 05-06· Lech Mazur suite data10.6%
3Gemini 2.5 Pro· Lech Mazur suite data10.8%
4Grok 3 Minihigh, beta· Lech Mazur suite data10.8%
5GLM 4.5· Lech Mazur suite data11.3%
6o1medium reasoning· Lech Mazur suite data11.7%
7Qwen3 30B A3B· Lech Mazur suite data12.3%
8Gemini 2.5 Pro· Lech Mazur suite data12.4%
9Grok 4· Lech Mazur suite data12.4%
10Gemini 2.0 Flash Experimentalthink· Lech Mazur suite data12.4%
11DeepSeek R1· Lech Mazur suite data12.7%
12O1 Preview· Lech Mazur suite data13%
13Claude Sonnet 4thinking, 16k· Lech Mazur suite data13.2%
14GPT-5 minimedium reasoning· Lech Mazur suite data13.3%
15Gemini 1.5 Pro· Lech Mazur suite data13.5%
16GPT-4.5 Preview· Lech Mazur suite data13.6%
17Grok 3 Minilow, beta· Lech Mazur suite data14%
18Grok 3no reasoning, beta· Lech Mazur suite data14.2%
19O3 Promedium reasoning· Lech Mazur suite data14.2%
20OpenAI o3high reasoning· Lech Mazur suite data14.4%
21Claude Sonnet 3.7thinking, 16k· Lech Mazur suite data14.7%
22Claude Sonnet 4no reasoning· Lech Mazur suite data14.9%
23GPT 4o· Lech Mazur suite data15.3%
24QwQ 32B16k· Lech Mazur suite data15.6%
25Qwen 3 235B A22B· Lech Mazur suite data15.6%
26GPT OSS 120Bmedium reasoning· Lech Mazur suite data15.7%
27o4 Minihigh reasoning· Lech Mazur suite data15.8%
28Claude Opus 4thinking, 16k· Lech Mazur suite data15.9%
29ERNIE 4.5 300B A47B· Lech Mazur suite data16%
30Gemini 2.5 Flash Preview24k· Lech Mazur suite data16.8%
31Claude Opus 4no reasoning· Lech Mazur suite data17.1%
32Claude Opus 4.1thinking, 16k· Lech Mazur suite data17.1%
33Llama 3.1 405B· Lech Mazur suite data17.6%
34o3 Minimedium reasoning· Lech Mazur suite data17.9%
35gemini-2.0-pro-exp-02-05· Lech Mazur suite data18.4%
36o3 Minihigh reasoning· Lech Mazur suite data18.4%
37Claude Opus 4.1no reasoning· Lech Mazur suite data18.5%
38O1 Mini· Lech Mazur suite data18.6%
39Qwen2.5 72B· Lech Mazur suite data19.1%
40Claude Sonnet 3.7· Lech Mazur suite data19.8%
41Claude 3.5 Sonnet· Lech Mazur suite data19.9%
42Kimi K2· Lech Mazur suite data20.4%
43Mistral Large 2· Lech Mazur suite data21.4%
44Qwen2.5-Max· Lech Mazur suite data21.8%
45Mistral Medium 3· Lech Mazur suite data21.9%
46Llama 4 Maverick· Lech Mazur suite data22.6%
47Claude 3 Opus· Lech Mazur suite data22.7%
48Llama 3.3 70B Instruct· Lech Mazur suite data22.8%
49MiniMax Text 01· Lech Mazur suite data23.9%
50Mistral-Small-3· Lech Mazur suite data25.2%

Showing 1–50 of 60 results.