Skip to content

Benchmark

LLM-AggreFact CNN

Factuality benchmarks measure how often models get facts right. Scores are percentages. Higher is better.

Models and systems measured
47
Scores recorded
47
Sources
1
IQ weight
No weight published

Score distribution

61.1%
Mean
63.6%
Median
6.2%
Std dev
51.5%
10th percentile
67.6%
90th percentile
49.4%
Min
49.4%70.1%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
AlignScoregrounded claim checker, CNN, renderer-v1· LLM-AggreFact data52.4%
Bespoke-Minicheck-7Bgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data65.5%
Claude-2.1grounded claim checker, CNN, renderer-v1· LLM-AggreFact data59.9%
Claude-3 Opusgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data65.2%
Claude-3.5 Sonnetgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data67.6%
Clefgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data67.3%
Clef Flashgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data61%
d1-ultragrounded claim checker, CNN, renderer-v1· LLM-AggreFact data63.9%
DAEgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data50.8%
Decider v1.2grounded claim checker, CNN, renderer-v1· LLM-AggreFact data65.5%
Drex v1.5grounded claim checker, CNN, renderer-v1· LLM-AggreFact data65.8%
FactCG-DeBERTa-Lgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data70.1%
Gemini-Progrounded claim checker, CNN, renderer-v1· LLM-AggreFact data49.4%
GPT-3.5-Turbogrounded claim checker, CNN, renderer-v1· LLM-AggreFact data63.2%
gpt-4-turbo-previewgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data66.7%
gpt-4o-2024-05-13grounded claim checker, CNN, renderer-v1· LLM-AggreFact data68.1%
gpt-4o-mini-2024-07-18grounded claim checker, CNN, renderer-v1· LLM-AggreFact data61.8%
Granite Guardian 3.3grounded claim checker, CNN, renderer-v1· LLM-AggreFact data67%
HHEM-2.1-opengrounded claim checker, CNN, renderer-v1· LLM-AggreFact data62.8%
InternLM2.5-7B-chatgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data58.4%
Jev 1.13.0grounded claim checker, CNN, renderer-v1· LLM-AggreFact data67.2%
Llama 3.2 1B InstructLlama-3.2-1B-Instruct, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data50.1%
Llama 3.2 3BLlama-3.2-3B-Instruct, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data51.5%
Llama 3.3 70B InstructLlama-3.3-70B-Instruct, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data68.7%
Llama-3-70B-Instructgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data63.7%
Llama-3.1-405B-Instructgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data64.8%
Llama-3.1-70B-Instructgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data65.7%
Llama-3.1-8B-Instructgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data54.7%
MiniCheck-DeBERTa-Lgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data64.2%
MiniCheck-Flan-T5-Lgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data69.9%
MiniCheck-RoBERTa-Lgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data63.7%
Mistral-8x7Bgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data55%
Mistral-Largegrounded claim checker, CNN, renderer-v1· LLM-AggreFact data58.4%
Mistral-Large 2grounded claim checker, CNN, renderer-v1· LLM-AggreFact data64.8%
Mixtral-8x22Bgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data57.3%
nimble-v4grounded claim checker, CNN, renderer-v1· LLM-AggreFact data67.8%
nimble-v4-maxgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data67.6%
PaLM2-Bisongrounded claim checker, CNN, renderer-v1· LLM-AggreFact data52.4%
Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data55.5%
Qwen2.5 72B InstructQwen2.5-72B-Instruct, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data63.6%
Qwen2.5 7B InstructQwen2.5-7B-Instruct, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data54.6%
QwQ 32B PreviewQwQ-32B-Preview, grounded claim checker, CNN, renderer-v1· LLM-AggreFact data57%
SummaC-CVgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data65.2%
SummaC-ZSgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data51.1%
T5-NLI-Mixedgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data54.6%
Tülu-3-70Bgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data58.3%
Tülu-3-8Bgrounded claim checker, CNN, renderer-v1· LLM-AggreFact data51%

Data credits: LLM-AggreFact, Tang et al. Public numeric grounded checker balanced accuracy; results licence not separately established.