Skip to content

Benchmark

FaithJudge Hallucination Rate Faithbench (Summarization)

Factuality benchmarks measure how often models get facts right. Scores are percentages. Lower is better.

Models and systems measured
43
Scores recorded
43
Sources
1
IQ weight
1.89

Score distribution

46.1%
Mean
44.4%
Median
14.6%
Std dev
33.3%
10th percentile
69.4%
90th percentile
19.4%
Min
19.4%88.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.89 for chat models, measured over 31 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Gemini 2.5 Flashgemini-2.5-flash, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data19.4%
2Gemini 2.5 Progemini-2.5-pro-exp-03-25, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data25%
3Gemini 2.5 Progemini-2.5-pro, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data25%
4Gemini 2.0 Flashgemini-2.0-flash-001, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data29.2%
5GPT 4.1gpt-4.1-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data36.1%
6Qwen2.5 32B InstructQwen2.5-32B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data36.1%
7GPT-4.1 minigpt-4.1-mini-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data37.5%
8GPT-4.5 Previewgpt-4.5-preview-2025-02-27, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data37.5%
9Claude Sonnet 3.7claude-3-7-sonnet-20250219, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data38.9%
10jamba-1.6-miniAI21-Jamba-mini-1.6, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data38.9%
11GPT 4ogpt-4o-2024-11-20, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data40.3%
12Grok 3grok-3, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data41.7%
13Mistral Small 24B InstructMistral-Small-24B-Instruct-2501, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization), 2501· FaithJudge data43.1%
14Qwen2.5 72B InstructQwen2.5-72B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data43.1%
15GPT-3.5 Turbogpt-3.5-turbo-0125, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data44.4%
16Llama 3.1 8B InstructLlama-3.1-8B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data44.4%
17Llama 3.3 70B InstructLlama-3.3-70B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data44.4%
18Phi 4phi-4, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data44.4%
19Claude Opus 4claude-opus-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data45.8%
20Claude Sonnet 4claude-sonnet-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data48.6%
21QwQ 32BQwQ-32B, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data50%
22GPT-4o minigpt-4o-mini-2024-07-18, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data51.4%
23Llama 4 Maverickllama-4-maverick, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data51.4%
24Qwen 2.5 14B InstructQwen2.5-14B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data54.2%
25Phi 4 MiniPhi-4-mini-instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data61.1%
26Qwen2.5 7B InstructQwen2.5-7B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data62.5%
27Qwen2.5 3B InstructQwen2.5-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data69.4%
28Llama 3.2 3BLlama-3.2-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data72.2%
29Llama 3.2 1B InstructLlama-3.2-1B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data75%
30Qwen2.5 1.5B InstructQwen2.5-1.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data84.7%
31Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, Faithbench (Summarization)· FaithJudge data88.9%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
claude-3-7-sonnet-thinking-20250219thinking, judge o3-mini-highthinking· FaithJudge data45.8%
claude-opus-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data38.9%
claude-sonnet-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data43.1%
o3-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data43.1%
o3-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data47.2%
o3-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data36.1%
o3-mini-high-2025-01-31reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data34.7%
o3-mini-low-2025-01-31reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data33.3%
o3-mini-medium-2025-01-31reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data33.3%
o4-mini-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data54.2%
o4-mini-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data44.4%
o4-mini-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data44.4%

Data credits: FaithJudge, Vectara; Tamber et al., EMNLP 2025 Industry. Public numeric README leaderboard; results licence not separately established.