Skip to content

Benchmark

HELM Lite: NarrativeQA - F1

Scores are percentages. Higher is better.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
0.96

Score distribution

69.5%
Mean
74.2%
Median
12.7%
Std dev
58.1%
10th percentile
78.3%
90th percentile
11.1%
Min
11.1%80.4%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.96 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 4o· Stanford HELM Lite data80.4%
2Meta Llama 3 70b· Stanford HELM Lite data79.8%
3DeepSeek V3· Stanford HELM Lite data79.6%
4Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data79.1%
5Nova Prov1· Stanford HELM Lite data79.1%
6Gemma 2 27B IT· Stanford HELM Lite data79%
7Gemini 2.0 Flash Experimental· Stanford HELM Lite data78.3%
8gemini-1.5-flash-001· Stanford HELM Lite data78.3%
9gemini-1.5-pro-001· Stanford HELM Lite data78.3%
10Yi-34B· Stanford HELM Lite data78.2%
11Mistral Large2407· Stanford HELM Lite data77.9%
12mixtral-8x22b· Stanford HELM Lite data77.9%
13Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data77.7%
14Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data77.2%
15Nova Litev1· Stanford HELM Lite data76.8%
16GPT-4o mini· Stanford HELM Lite data76.8%
17GPT 4· Stanford HELM Lite data76.8%
18Gemma 2 9B IT· Stanford HELM Lite data76.8%
19Llama 2 70b· Stanford HELM Lite data76.3%
20Claude Haiku 3.5· Stanford HELM Lite data76.3%
21GPT-4 Turbo· Stanford HELM Lite data76.1%
22Llama 3.1 8B Instruct Turbo· Stanford HELM Lite data75.6%
23Llama-3.2-11B-Vision-Instruct-Turbo· Stanford HELM Lite data75.6%
24gemini-1.5-pro-002· Stanford HELM Lite data75.6%
25Phi-3 Small 8K Instruct· Stanford HELM Lite data75.4%
26Meta Llama 3 8b· Stanford HELM Lite data75.4%
27gemma-7b· Stanford HELM Lite data75.2%
28Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data74.9%
29command· Stanford HELM Lite data74.9%
30Claude 3.5 Sonnet· Stanford HELM Lite data74.6%
31Jamba 1.5 Mini· Stanford HELM Lite data74.6%
32gemini-1.5-flash-002· Stanford HELM Lite data74.6%
33Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data74.5%
34Nova Microv1· Stanford HELM Lite data74.4%
35Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data74.2%
36Command R+· Stanford HELM Lite data74.2%
37Llama 2 13b· Stanford HELM Lite data74.1%
38Command R Plus· Stanford HELM Lite data73.5%
39Open Mistral Nemo· Stanford HELM Lite data73.1%
40Text Davinci 003· Stanford HELM Lite data73.1%
41Qwen2 72B Instruct· Stanford HELM Lite data72.7%
42GPT 4 1106 Preview· Stanford HELM Lite data72.7%
43Phi-3 Medium 4K Instruct· Stanford HELM Lite data72.4%
44Qwen1.5-110B-Chat· Stanford HELM Lite data72.1%
45Text Davinci 002· Stanford HELM Lite data71.9%
46Claude 2· Stanford HELM Lite data71.8%
47Mistral 7B v0.1· Stanford HELM Lite data71.6%
48Mistral (7B) Instruct v0.3· Stanford HELM Lite data71.6%
49Qwen1.5-14B· Stanford HELM Lite data71.1%
50phi-2· Stanford HELM Lite data70.3%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)