Skip to content

Benchmark

LLM-AggreFact ExpertQA

Factuality benchmarks measure how often models get facts right. Scores are percentages. Higher is better.

Models and systems measured
47
Scores recorded
47
Sources
1
IQ weight
No weight published

Score distribution

58.2%
Mean
58.5%
Median
2.1%
Std dev
55.7%
10th percentile
60.2%
90th percentile
49.9%
Min
49.9%61%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
AlignScoregrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.3%
Bespoke-Minicheck-7Bgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.2%
Claude-2.1grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.8%
Claude-3 Opusgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.8%
Claude-3.5 Sonnetgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.9%
Clefgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59%
Clef Flashgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.2%
d1-ultragrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.2%
DAEgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data56.2%
Decider v1.2grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.6%
Drex v1.5grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.3%
FactCG-DeBERTa-Lgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.1%
Gemini-Progrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data56.8%
GPT-3.5-Turbogrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data57.2%
gpt-4-turbo-previewgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.2%
gpt-4o-2024-05-13grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.6%
gpt-4o-mini-2024-07-18grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.3%
Granite Guardian 3.3grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.6%
HHEM-2.1-opengrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.4%
InternLM2.5-7B-chatgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data55.2%
Jev 1.13.0grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59.9%
Llama 3.2 1B InstructLlama-3.2-1B-Instruct, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data49.9%
Llama 3.2 3BLlama-3.2-3B-Instruct, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data55.8%
Llama 3.3 70B InstructLlama-3.3-70B-Instruct, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.3%
Llama-3-70B-Instructgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data57.8%
Llama-3.1-405B-Instructgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.5%
Llama-3.1-70B-Instructgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.5%
Llama-3.1-8B-Instructgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data57.8%
MiniCheck-DeBERTa-Lgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data58.9%
MiniCheck-Flan-T5-Lgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data59%
MiniCheck-RoBERTa-Lgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data57.4%
Mistral-8x7Bgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data56.3%
Mistral-Largegrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60%
Mistral-Large 2grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.8%
Mixtral-8x22Bgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data57.4%
nimble-v4grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.1%
nimble-v4-maxgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.4%
PaLM2-Bisongrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data56.6%
Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data56.8%
Qwen2.5 72B InstructQwen2.5-72B-Instruct, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60.1%
Qwen2.5 7B InstructQwen2.5-7B-Instruct, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data61%
QwQ 32B PreviewQwQ-32B-Preview, grounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data60%
SummaC-CVgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data54.9%
SummaC-ZSgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data55.2%
T5-NLI-Mixedgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data55.6%
Tülu-3-70Bgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data55.7%
Tülu-3-8Bgrounded claim checker, ExpertQA, renderer-v1· LLM-AggreFact data56.2%

Data credits: LLM-AggreFact, Tang et al. Public numeric grounded checker balanced accuracy; results licence not separately established.