Skip to content

Benchmark

HELM Lite: NaturalQuestions (open-book) - F1

Scores are percentages. Higher is better.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
0.67

Score distribution

68.7%
Mean
72.3%
Median
12.3%
Std dev
60.3%
10th percentile
77.6%
90th percentile
7.2%
Min
7.2%82.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.67 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Nova Prov1· Stanford HELM Lite data82.9%
2Nova Litev1· Stanford HELM Lite data81.5%
3GPT 4o· Stanford HELM Lite data80.3%
4GPT-4 Turbo· Stanford HELM Lite data79.5%
5GPT 4· Stanford HELM Lite data79%
6Nova Microv1· Stanford HELM Lite data77.9%
7command· Stanford HELM Lite data77.7%
8Qwen2 72B Instruct· Stanford HELM Lite data77.6%
9Yi-34B· Stanford HELM Lite data77.5%
10Qwen1.5-14B· Stanford HELM Lite data77.2%
11Text Davinci 003· Stanford HELM Lite data77%
12DeepSeek V3· Stanford HELM Lite data76.5%
13GPT 4 1106 Preview· Stanford HELM Lite data76.3%
14Qwen1.5-72B· Stanford HELM Lite data75.8%
15Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data75.6%
16Claude 3.5 Sonnet· Stanford HELM Lite data74.9%
17Qwen1.5-7B· Stanford HELM Lite data74.9%
18gemini-1.5-pro-001· Stanford HELM Lite data74.8%
19Yi-6B· Stanford HELM Lite data74.8%
20GPT-4o mini· Stanford HELM Lite data74.6%
21Meta Llama 3 70b· Stanford HELM Lite data74.3%
22Qwen1.5-110B-Chat· Stanford HELM Lite data73.9%
23Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data73.9%
24Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data73.8%
25Gemma 2 9B IT· Stanford HELM Lite data73.8%
26Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data73.7%
27Mistral Large2407· Stanford HELM Lite data73.4%
28DeepSeek LLM 67B Chat· Stanford HELM Lite data73.3%
29Gemma 2 27B IT· Stanford HELM Lite data73.1%
30Claude Instant 1.2· Stanford HELM Lite data73.1%
31Phi-3 Medium 4K Instruct· Stanford HELM Lite data72.9%
32mixtral-8x22b· Stanford HELM Lite data72.6%
33gemini-1.5-pro-002· Stanford HELM Lite data72.6%
34Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data72.5%
35gemini-1.5-flash-001· Stanford HELM Lite data72.3%
36Gemini 2.0 Flash Experimental· Stanford HELM Lite data72.2%
37Command R+· Stanford HELM Lite data72%
38Jamba 1.5 Large· Stanford HELM Lite data71.8%
39gemini-1.5-flash-002· Stanford HELM Lite data71.8%
40Command R Plus· Stanford HELM Lite data71.1%
41Jamba 1.5 Mini· Stanford HELM Lite data71%
42Text Davinci 002· Stanford HELM Lite data71%
43Mistral 7B v0.1· Stanford HELM Lite data68.7%
44Command Light· Stanford HELM Lite data68.6%
45Meta Llama 3 8b· Stanford HELM Lite data68.1%
46phi-2· Stanford HELM Lite data68%
47Mistral (7B) Instruct v0.3· Stanford HELM Lite data68%
48GPT-3.5 Turbo· Stanford HELM Lite data67.8%
49Llama 3.1 8B Instruct Turbo· Stanford HELM Lite data67.7%
50Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data67.6%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)