Skip to content

Benchmark

HELM Lite: NaturalQuestions (closed-book) - F1

Scores are percentages. Higher is better.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
1.15

Score distribution

36%
Mean
37.5%
Median
9.1%
Std dev
25.3%
10th percentile
45.7%
90th percentile
2.8%
Min
2.8%50.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.15 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude 3.5 Sonnet· Stanford HELM Lite data50.2%
2GPT 4o· Stanford HELM Lite data50.1%
3GPT-4 Turbo· Stanford HELM Lite data48.2%
4mixtral-8x22b· Stanford HELM Lite data47.8%
5Meta Llama 3 70b· Stanford HELM Lite data47.5%
6DeepSeek V3· Stanford HELM Lite data46.7%
7Llama 2 70b· Stanford HELM Lite data46%
8GPT 4· Stanford HELM Lite data45.7%
9Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data45.7%
10Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data45.6%
11gemini-1.5-pro-002· Stanford HELM Lite data45.5%
12Mistral Large2407· Stanford HELM Lite data45.3%
13Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data45.2%
14Yi-34B· Stanford HELM Lite data44.3%
15Gemini 2.0 Flash Experimental· Stanford HELM Lite data44.3%
16Claude 3 Opus· Stanford HELM Lite data44.1%
17text-unicorn· Stanford HELM Lite data43.5%
18GPT 4 1106 Preview· Stanford HELM Lite data43.5%
19Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data43.1%
20Claude 2· Stanford HELM Lite data42.8%
21Qwen1.5-72B· Stanford HELM Lite data41.7%
22Text Davinci 003· Stanford HELM Lite data41.3%
23DeepSeek LLM 67B Chat· Stanford HELM Lite data41.2%
24Nova Prov1· Stanford HELM Lite data40.5%
25Text Davinci 002· Stanford HELM Lite data39.4%
26Jamba 1.5 Large· Stanford HELM Lite data39.4%
27command· Stanford HELM Lite data39.1%
28Snowflake Arctic Instruct· Stanford HELM Lite data39%
29Qwen2 72B Instruct· Stanford HELM Lite data39%
30Jamba 1.5 Mini· Stanford HELM Lite data38.8%
31GPT-4o mini· Stanford HELM Lite data38.6%
32Jamba Instruct· Stanford HELM Lite data38.4%
33Meta Llama 3 8b· Stanford HELM Lite data37.8%
34gemini-1.5-pro-001· Stanford HELM Lite data37.8%
35Claude 2.1· Stanford HELM Lite data37.5%
36Llama 2 13b· Stanford HELM Lite data37.1%
37Mistral 7B v0.1· Stanford HELM Lite data36.7%
38Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data35.9%
39Gemma 2 27B IT· Stanford HELM Lite data35.3%
40Nova Litev1· Stanford HELM Lite data35.2%
41Command R+· Stanford HELM Lite data35.2%
42Qwen1.5-110B-Chat· Stanford HELM Lite data35%
43Claude Haiku 3.5· Stanford HELM Lite data34.4%
44Claude Instant 1.2· Stanford HELM Lite data34.3%
45Command R Plus· Stanford HELM Lite data34.3%
46gemma-7b· Stanford HELM Lite data33.6%
47GPT-3.5 Turbo· Stanford HELM Lite data33.5%
48Llama 2 7b· Stanford HELM Lite data33.3%
49gemini-1.5-flash-001· Stanford HELM Lite data33.2%
50Gemma 2 9B IT· Stanford HELM Lite data32.8%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)