Skip to content

Benchmark

HELM Lite: OpenbookQA - EM

Scores are percentages.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
2.13

Score distribution

85.8%
Mean
89.4%
Median
12.4%
Std dev
77.4%
10th percentile
95.4%
90th percentile
22.2%
Min
22.2%97.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.13 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude 3.5 Sonnet· Stanford HELM Lite data97.2%
2GPT-4 Turbo· Stanford HELM Lite data97%
3GPT 4o· Stanford HELM Lite data96.6%
4Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data96.2%
5GPT 4· Stanford HELM Lite data96%
6Nova Prov1· Stanford HELM Lite data96%
7Claude 3 Opus· Stanford HELM Lite data95.6%
8DeepSeek V3· Stanford HELM Lite data95.4%
9Qwen2 72B Instruct· Stanford HELM Lite data95.4%
10gemini-1.5-pro-002· Stanford HELM Lite data95.2%
11GPT 4 1106 Preview· Stanford HELM Lite data95%
12Jamba 1.5 Large· Stanford HELM Lite data94.8%
13Gemini 2.0 Flash Experimental· Stanford HELM Lite data94.6%
14Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data94.2%
15Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data94%
16Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data93.8%
17text-unicorn· Stanford HELM Lite data93.8%
18Meta Llama 3 70b· Stanford HELM Lite data93.4%
19Mistral Large2407· Stanford HELM Lite data93.2%
20Qwen1.5-72B· Stanford HELM Lite data93%
21gemini-1.5-flash-001· Stanford HELM Lite data92.8%
22Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data92.8%
23Nova Litev1· Stanford HELM Lite data92.8%
24Qwen1.5-110B-Chat· Stanford HELM Lite data92.2%
25GPT-4o mini· Stanford HELM Lite data92%
26Yi-34B· Stanford HELM Lite data92%
27Claude 3 Sonnet· Stanford HELM Lite data91.8%
28Gemma 2 27B IT· Stanford HELM Lite data91.8%
29Phi-3 Medium 4K Instruct· Stanford HELM Lite data91.6%
30gemini-1.5-flash-002· Stanford HELM Lite data91.4%
31Phi-3 Small 8K Instruct· Stanford HELM Lite data91.2%
32DBRX-instruct· Stanford HELM Lite data91%
33Gemma 2 9B IT· Stanford HELM Lite data91%
34gemini-1.5-pro-001· Stanford HELM Lite data90.2%
35Mistral Large· Stanford HELM Lite data89.4%
36Jamba 1.5 Mini· Stanford HELM Lite data89%
37Nova Microv1· Stanford HELM Lite data88.8%
38mixtral-8x22b· Stanford HELM Lite data88.2%
39DeepSeek LLM 67B Chat· Stanford HELM Lite data88%
40Claude 2.1· Stanford HELM Lite data87.2%
41Claude 2· Stanford HELM Lite data86.2%
42Mistral Small· Stanford HELM Lite data86.2%
43Qwen1.5-14B· Stanford HELM Lite data86.2%
44Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data86.2%
45Claude Haiku 3.5· Stanford HELM Lite data85.4%
46Claude Instant 1.2· Stanford HELM Lite data84.4%
47Claude 3 Haiku· Stanford HELM Lite data83.8%
48GPT-3.5 Turbo· Stanford HELM Lite data83.8%
49Llama 2 70b· Stanford HELM Lite data83.8%
50Command R Plus· Stanford HELM Lite data82.8%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)