Skip to content

Benchmark

FaithJudge Hallucination Rate

Factuality benchmarks measure how often models get facts right. Scores are percentages. Lower is better.

Models and systems measured
43
Scores recorded
43
Sources
1
IQ weight
2.07

Score distribution

24.1%
Mean
18.6%
Median
16.3%
Std dev
11.6%
10th percentile
46.2%
90th percentile
6.3%
Min
6.3%76.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.07 for chat models, measured over 31 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Gemini 2.5 Flashgemini-2.5-flash, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data6.3%
2Gemini 2.5 Progemini-2.5-pro, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data6.7%
3Gemini 2.5 Progemini-2.5-pro-exp-03-25, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data7.6%
4Gemini 2.0 Flashgemini-2.0-flash-001, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data10.2%
5GPT 4.1gpt-4.1-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data11.9%
6GPT-4.5 Previewgpt-4.5-preview-2025-02-27, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data11.9%
7GPT-3.5 Turbogpt-3.5-turbo-0125, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data14.9%
8Grok 3grok-3, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data15.3%
9GPT-4.1 minigpt-4.1-mini-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data15.7%
10GPT 4ogpt-4o-2024-11-20, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data15.9%
11Claude Sonnet 3.7claude-3-7-sonnet-20250219, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data16.1%
12Llama 3.3 70B InstructLlama-3.3-70B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data16.4%
13Mistral Small 24B InstructMistral-Small-24B-Instruct-2501, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall, 2501· FaithJudge data17%
14Phi 4phi-4, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data17%
15Claude Sonnet 4claude-sonnet-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data18.6%
16GPT-4o minigpt-4o-mini-2024-07-18, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data18.6%
17Qwen2.5 32B InstructQwen2.5-32B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data19.2%
18Claude Opus 4claude-opus-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data19.6%
19Llama 4 Maverickllama-4-maverick, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data20.6%
20Qwen2.5 72B InstructQwen2.5-72B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data20.7%
21QwQ 32BQwQ-32B, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data24.7%
22Llama 3.1 8B InstructLlama-3.1-8B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data28.4%
23Qwen 2.5 14B InstructQwen2.5-14B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data29%
24Phi 4 MiniPhi-4-mini-instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data38.4%
25Qwen2.5 7B InstructQwen2.5-7B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data38.5%
26jamba-1.6-miniAI21-Jamba-mini-1.6, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data39.1%
27Llama 3.2 3BLlama-3.2-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data46.2%
28Qwen2.5 3B InstructQwen2.5-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data56%
29Qwen2.5 1.5B InstructQwen2.5-1.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data66.7%
30Llama 3.2 1B InstructLlama-3.2-1B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data67.7%
31Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, overall· FaithJudge data76.3%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
claude-3-7-sonnet-thinking-20250219thinking, judge o3-mini-highthinking· FaithJudge data16.2%
claude-opus-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data14.1%
claude-sonnet-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data18.2%
o3-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data18.6%
o3-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data20.6%
o3-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data17.8%
o3-mini-high-2025-01-31reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data12.5%
o3-mini-low-2025-01-31reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data11.9%
o3-mini-medium-2025-01-31reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data11.6%
o4-mini-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data29.9%
o4-mini-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data28%
o4-mini-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data25.8%

Data credits: FaithJudge, Vectara; Tamber et al., EMNLP 2025 Industry. Public numeric README leaderboard; results licence not separately established.