Skip to content

Benchmark

FaithJudge Hallucination Rate RagTruth (Data-to-Text Writing)

Factuality benchmarks measure how often models get facts right. Scores are percentages. Lower is better.

Models and systems measured
43
Scores recorded
43
Sources
1
IQ weight
2.41

Score distribution

30.6%
Mean
21.3%
Median
25.7%
Std dev
8%
10th percentile
81.3%
90th percentile
4%
Min
4%90%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.41 for chat models, measured over 31 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Gemini 2.5 Progemini-2.5-pro-exp-03-25, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data4%
2Gemini 2.5 Flashgemini-2.5-flash, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data4.7%
3Gemini 2.5 Progemini-2.5-pro, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data4.7%
4GPT 4.1gpt-4.1-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data8%
5GPT-4.5 Previewgpt-4.5-preview-2025-02-27, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data8%
6Claude Sonnet 3.7claude-3-7-sonnet-20250219, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data12.7%
7Gemini 2.0 Flashgemini-2.0-flash-001, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data13.3%
8Grok 3grok-3, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data14%
9GPT-3.5 Turbogpt-3.5-turbo-0125, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data15.3%
10Mistral Small 24B InstructMistral-Small-24B-Instruct-2501, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing), 2501· FaithJudge data18%
11GPT 4ogpt-4o-2024-11-20, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data20%
12Qwen2.5 72B InstructQwen2.5-72B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data20%
13GPT-4.1 minigpt-4.1-mini-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data21.3%
14GPT-4o minigpt-4o-mini-2024-07-18, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data21.3%
15Claude Sonnet 4claude-sonnet-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data22%
16Llama 3.3 70B InstructLlama-3.3-70B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data22%
17Claude Opus 4claude-opus-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data22.7%
18Llama 4 Maverickllama-4-maverick, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data23.3%
19QwQ 32BQwQ-32B, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data24%
20Phi 4phi-4, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data24.7%
21Qwen2.5 32B InstructQwen2.5-32B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data28.7%
22Qwen 2.5 14B InstructQwen2.5-14B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data48.7%
23Llama 3.1 8B InstructLlama-3.1-8B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data51.3%
24Qwen2.5 7B InstructQwen2.5-7B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data60.7%
25jamba-1.6-miniAI21-Jamba-mini-1.6, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data61.3%
26Phi 4 MiniPhi-4-mini-instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data75.3%
27Llama 3.2 3BLlama-3.2-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data81.3%
28Llama 3.2 1B InstructLlama-3.2-1B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data88.7%
29Qwen2.5 1.5B InstructQwen2.5-1.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data88.7%
30Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data89.3%
31Qwen2.5 3B InstructQwen2.5-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Data-to-Text Writing)· FaithJudge data90%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
claude-3-7-sonnet-thinking-20250219thinking, judge o3-mini-highthinking· FaithJudge data14%
claude-opus-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data13.3%
claude-sonnet-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data19.3%
o3-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data18.7%
o3-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data18%
o3-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data15.3%
o3-mini-high-2025-01-31reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data12%
o3-mini-low-2025-01-31reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data11.3%
o3-mini-medium-2025-01-31reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data8%
o4-mini-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data36%
o4-mini-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data28.7%
o4-mini-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data32.7%

Data credits: FaithJudge, Vectara; Tamber et al., EMNLP 2025 Industry. Public numeric README leaderboard; results licence not separately established.