Skip to content

Benchmark

LLM-AggreFact XSum

Factuality benchmarks measure how often models get facts right. Scores are percentages. Higher is better.

Models and systems measured
47
Scores recorded
47
Sources
1
IQ weight
No weight published

Score distribution

69.8%
Mean
72.4%
Median
7.5%
Std dev
59%
10th percentile
77.4%
90th percentile
50.9%
Min
50.9%78.6%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
AlignScoregrounded claim checker, XSum, renderer-v1· LLM-AggreFact data71.4%
Bespoke-Minicheck-7Bgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data77.8%
Claude-2.1grounded claim checker, XSum, renderer-v1· LLM-AggreFact data66.4%
Claude-3 Opusgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data72.4%
Claude-3.5 Sonnetgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data75.1%
Clefgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data75.7%
Clef Flashgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data77.1%
d1-ultragrounded claim checker, XSum, renderer-v1· LLM-AggreFact data77.6%
DAEgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data59.1%
Decider v1.2grounded claim checker, XSum, renderer-v1· LLM-AggreFact data77.9%
Drex v1.5grounded claim checker, XSum, renderer-v1· LLM-AggreFact data77.4%
FactCG-DeBERTa-Lgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data73.9%
Gemini-Progrounded claim checker, XSum, renderer-v1· LLM-AggreFact data60.6%
GPT-3.5-Turbogrounded claim checker, XSum, renderer-v1· LLM-AggreFact data72.4%
gpt-4-turbo-previewgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data76.5%
gpt-4o-2024-05-13grounded claim checker, XSum, renderer-v1· LLM-AggreFact data76.8%
gpt-4o-mini-2024-07-18grounded claim checker, XSum, renderer-v1· LLM-AggreFact data73.6%
Granite Guardian 3.3grounded claim checker, XSum, renderer-v1· LLM-AggreFact data74.9%
HHEM-2.1-opengrounded claim checker, XSum, renderer-v1· LLM-AggreFact data67.8%
InternLM2.5-7B-chatgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data57.9%
Jev 1.13.0grounded claim checker, XSum, renderer-v1· LLM-AggreFact data75.5%
Llama 3.2 1B InstructLlama-3.2-1B-Instruct, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data50.9%
Llama 3.2 3BLlama-3.2-3B-Instruct, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data60.5%
Llama 3.3 70B InstructLlama-3.3-70B-Instruct, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data74.7%
Llama-3-70B-Instructgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data70.2%
Llama-3.1-405B-Instructgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data75.1%
Llama-3.1-70B-Instructgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data72.5%
Llama-3.1-8B-Instructgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data68.5%
MiniCheck-DeBERTa-Lgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data71%
MiniCheck-Flan-T5-Lgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data74.3%
MiniCheck-RoBERTa-Lgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data70.8%
Mistral-8x7Bgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data65.5%
Mistral-Largegrounded claim checker, XSum, renderer-v1· LLM-AggreFact data76.3%
Mistral-Large 2grounded claim checker, XSum, renderer-v1· LLM-AggreFact data74.7%
Mixtral-8x22Bgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data70.3%
nimble-v4grounded claim checker, XSum, renderer-v1· LLM-AggreFact data78.5%
nimble-v4-maxgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data78.6%
PaLM2-Bisongrounded claim checker, XSum, renderer-v1· LLM-AggreFact data59%
Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data62.9%
Qwen2.5 72B InstructQwen2.5-72B-Instruct, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data73%
Qwen2.5 7B InstructQwen2.5-7B-Instruct, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data69.9%
QwQ 32B PreviewQwQ-32B-Preview, grounded claim checker, XSum, renderer-v1· LLM-AggreFact data71.6%
SummaC-CVgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data54.5%
SummaC-ZSgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data61.5%
T5-NLI-Mixedgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data52.3%
Tülu-3-70Bgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data66.4%
Tülu-3-8Bgrounded claim checker, XSum, renderer-v1· LLM-AggreFact data58.5%

Data credits: LLM-AggreFact, Tang et al. Public numeric grounded checker balanced accuracy; results licence not separately established.