Skip to content

Benchmark

LLM-AggreFact ClaimVerify

Factuality benchmarks measure how often models get facts right. Scores are percentages. Higher is better.

Models and systems measured
47
Scores recorded
47
Sources
1
IQ weight
No weight published

Score distribution

69%
Mean
69.7%
Median
5.9%
Std dev
61.8%
10th percentile
75.9%
90th percentile
50.5%
Min
50.5%78.6%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
AlignScoregrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data69.6%
Bespoke-Minicheck-7Bgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data75.3%
Claude-2.1grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data69.7%
Claude-3 Opusgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data69.3%
Claude-3.5 Sonnetgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data71.4%
Clefgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data78.6%
Clef Flashgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data74.5%
d1-ultragrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data76.8%
DAEgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data64%
Decider v1.2grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data75.8%
Drex v1.5grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data70.8%
FactCG-DeBERTa-Lgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data78.5%
Gemini-Progrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data61.8%
GPT-3.5-Turbogrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data65.2%
gpt-4-turbo-previewgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data67.6%
gpt-4o-2024-05-13grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data69%
gpt-4o-mini-2024-07-18grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data69.8%
Granite Guardian 3.3grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data75.9%
HHEM-2.1-opengrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data73.9%
InternLM2.5-7B-chatgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data59.8%
Jev 1.13.0grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data76.9%
Llama 3.2 1B InstructLlama-3.2-1B-Instruct, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data50.5%
Llama 3.2 3BLlama-3.2-3B-Instruct, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data62.3%
Llama 3.3 70B InstructLlama-3.3-70B-Instruct, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data67.4%
Llama-3-70B-Instructgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data67.8%
Llama-3.1-405B-Instructgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data67.5%
Llama-3.1-70B-Instructgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data70.3%
Llama-3.1-8B-Instructgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data66.5%
MiniCheck-DeBERTa-Lgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data75.6%
MiniCheck-Flan-T5-Lgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data74.6%
MiniCheck-RoBERTa-Lgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data77.4%
Mistral-8x7Bgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data64.3%
Mistral-Largegrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data67.6%
Mistral-Large 2grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data71.8%
Mixtral-8x22Bgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data63.8%
nimble-v4grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data71.4%
nimble-v4-maxgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data75.9%
PaLM2-Bisongrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data60.5%
Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data61.4%
Qwen2.5 72B InstructQwen2.5-72B-Instruct, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data70%
Qwen2.5 7B InstructQwen2.5-7B-Instruct, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data70.1%
QwQ 32B PreviewQwQ-32B-Preview, grounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data67.7%
SummaC-CVgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data70.9%
SummaC-ZSgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data69.7%
T5-NLI-Mixedgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data59.5%
Tülu-3-70Bgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data62.2%
Tülu-3-8Bgrounded claim checker, ClaimVerify, renderer-v1· LLM-AggreFact data63.2%

Data credits: LLM-AggreFact, Tang et al. Public numeric grounded checker balanced accuracy; results licence not separately established.