Skip to content

Benchmark

LLM-AggreFact WiCE

Factuality benchmarks measure how often models get facts right. Scores are percentages. Higher is better.

Models and systems measured
47
Scores recorded
47
Sources
1
IQ weight
No weight published

Score distribution

72.2%
Mean
73.9%
Median
9.3%
Std dev
58.7%
10th percentile
83.9%
90th percentile
49.7%
Min
49.7%87.1%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
AlignScoregrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data66%
Bespoke-Minicheck-7Bgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data83%
Claude-2.1grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data64.3%
Claude-3 Opusgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data75%
Claude-3.5 Sonnetgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data77.7%
Clefgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data85.8%
Clef Flashgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data81.7%
d1-ultragrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data82.7%
DAEgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data58.5%
Decider v1.2grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data87.1%
Drex v1.5grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data84.4%
FactCG-DeBERTa-Lgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data74.2%
Gemini-Progrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data65.8%
GPT-3.5-Turbogrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data68.5%
gpt-4-turbo-previewgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data80.4%
gpt-4o-2024-05-13grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data78.5%
gpt-4o-mini-2024-07-18grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data76.3%
Granite Guardian 3.3grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data76.6%
HHEM-2.1-opengrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data75.9%
InternLM2.5-7B-chatgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data63.7%
Jev 1.13.0grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data83.9%
Llama 3.2 1B InstructLlama-3.2-1B-Instruct, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data49.7%
Llama 3.2 3BLlama-3.2-3B-Instruct, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data58.5%
Llama 3.3 70B InstructLlama-3.3-70B-Instruct, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data76.6%
Llama-3-70B-Instructgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data74.4%
Llama-3.1-405B-Instructgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data71.8%
Llama-3.1-70B-Instructgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data73.9%
Llama-3.1-8B-Instructgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data72%
MiniCheck-DeBERTa-Lgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data69.4%
MiniCheck-Flan-T5-Lgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data72.2%
MiniCheck-RoBERTa-Lgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data67.6%
Mistral-8x7Bgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data63.8%
Mistral-Largegrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data76.6%
Mistral-Large 2grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data80.3%
Mixtral-8x22Bgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data69.5%
nimble-v4grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data85.1%
nimble-v4-maxgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data87.1%
PaLM2-Bisongrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data63.4%
Qwen2.5 0.5b InstructQwen2.5-0.5B-Instruct, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data65.9%
Qwen2.5 72B InstructQwen2.5-72B-Instruct, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data80.2%
Qwen2.5 7B InstructQwen2.5-7B-Instruct, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data76.3%
QwQ 32B PreviewQwQ-32B-Preview, grounded claim checker, WiCE, renderer-v1· LLM-AggreFact data72.3%
SummaC-CVgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data54.3%
SummaC-ZSgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data62.8%
T5-NLI-Mixedgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data55.3%
Tülu-3-70Bgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data64.6%
Tülu-3-8Bgrounded claim checker, WiCE, renderer-v1· LLM-AggreFact data58.7%

Data credits: LLM-AggreFact, Tang et al. Public numeric grounded checker balanced accuracy; results licence not separately established.