Skip to content

Benchmark

FaithJudge Hallucination Rate RagTruth (Question-Answering)

Factuality benchmarks measure how often models get facts right. Scores are percentages. Lower is better.

Models and systems measured
43
Scores recorded
43
Sources
1
IQ weight
1.93

Score distribution

11.4%
Mean
6.5%
Median
11.9%
Std dev
3.6%
10th percentile
27.3%
90th percentile
0.72%
Min
0.72%58.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.93 for chat models, measured over 31 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Gemini 2.0 Flashgemini-2.0-flash-001, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data0.72%
2Gemini 2.5 Progemini-2.5-pro-exp-03-25, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data0.72%
3Gemini 2.5 Progemini-2.5-pro, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data1.4%
4Gemini 2.5 Flashgemini-2.5-flash, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data2.9%
5GPT-4.1 minigpt-4.1-mini-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data3.6%
6GPT 4.1gpt-4.1-2025-04-14, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data4.3%
7Llama 3.3 70B InstructLlama-3.3-70B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data4.3%
8Phi 4phi-4, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data4.3%
9Claude Sonnet 4claude-sonnet-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data5%
10GPT 4ogpt-4o-2024-11-20, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data5%
11GPT-4.5 Previewgpt-4.5-preview-2025-02-27, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data5%
12Phi 4 MiniPhi-4-mini-instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data5%
13GPT-3.5 Turbogpt-3.5-turbo-0125, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data5.8%
14GPT-4o minigpt-4o-mini-2024-07-18, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data6.5%
15Grok 3grok-3, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data6.5%
16Qwen 2.5 14B InstructQwen2.5-14B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data6.5%
17Qwen2.5 32B InstructQwen2.5-32B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data6.5%
18QwQ 32BQwQ-32B, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data6.5%
19Claude Sonnet 3.7claude-3-7-sonnet-20250219, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data9.4%
20Llama 4 Maverickllama-4-maverick, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data9.4%
21Claude Opus 4claude-opus-4-20250514, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data10.1%
22Mistral Small 24B InstructMistral-Small-24B-Instruct-2501, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering), 2501· FaithJudge data10.1%
23Llama 3.1 8B InstructLlama-3.1-8B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data12.2%
24Llama 3.2 3BLlama-3.2-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data13.7%
25Qwen2.5 7B InstructQwen2.5-7B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data15.8%
26Qwen2.5 72B InstructQwen2.5-72B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data18.7%
27Qwen2.5 3B InstructQwen2.5-3B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data27.3%
28jamba-1.6-miniAI21-Jamba-mini-1.6, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data30.2%
29Qwen2.5 1.5B InstructQwen2.5-1.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data40.3%
30Llama 3.2 1B InstructLlama-3.2-1B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data44.6%
31Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, o3-mini-high, FaithJudge RAG generation hallucination rate; human-example-guided LLM judge, RagTruth (Question-Answering)· FaithJudge data58.3%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
claude-3-7-sonnet-thinking-20250219thinking, judge o3-mini-highthinking· FaithJudge data9.4%
claude-opus-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data5%
claude-sonnet-4-thinking-20250514thinking, judge o3-mini-highthinking· FaithJudge data7.2%
o3-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data3.6%
o3-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data7.2%
o3-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data9.4%
o3-mini-high-2025-01-31reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data6.5%
o3-mini-low-2025-01-31reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data7.2%
o3-mini-medium-2025-01-31reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data6.5%
o4-mini-high-2025-04-16reasoning effort high, judge o3-mini-highreasoning effort high· FaithJudge data18%
o4-mini-low-2025-04-16reasoning effort low, judge o3-mini-highreasoning effort low· FaithJudge data16.6%
o4-mini-medium-2025-04-16reasoning effort medium, judge o3-mini-highreasoning effort medium· FaithJudge data11.5%

Data credits: FaithJudge, Vectara; Tamber et al., EMNLP 2025 Industry. Public numeric README leaderboard; results licence not separately established.