Skip to content

Benchmark

HELM Lite: MMLU - EM

Scores are percentages. Higher is better.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
1.95

Score distribution

64%
Mean
65%
Median
9.6%
Std dev
51.8%
10th percentile
75.9%
90th percentile
30.5%
Min
30.5%80.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.95 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1DeepSeek V3· Stanford HELM Lite data80.3%
2Claude 3.5 Sonnet· Stanford HELM Lite data79.9%
3gemini-1.5-pro-002· Stanford HELM Lite data79.5%
4gemini-1.5-pro-001· Stanford HELM Lite data77.2%
5Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data77%
6Qwen2 72B Instruct· Stanford HELM Lite data76.9%
7Claude 3 Opus· Stanford HELM Lite data76.8%
8Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data75.9%
9Nova Prov1· Stanford HELM Lite data75.8%
10GPT 4o· Stanford HELM Lite data74.8%
11GPT 4· Stanford HELM Lite data73.5%
12Mistral Large2407· Stanford HELM Lite data72.5%
13Gemini 2.0 Flash Experimental· Stanford HELM Lite data71.7%
14GPT-4 Turbo· Stanford HELM Lite data71.1%
15Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data70.9%
16Qwen1.5-110B-Chat· Stanford HELM Lite data70.4%
17Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data70.3%
18gemini-1.5-flash-001· Stanford HELM Lite data70.3%
19text-unicorn· Stanford HELM Lite data70.2%
20mixtral-8x22b· Stanford HELM Lite data70.1%
21Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data70%
22GPT 4 1106 Preview· Stanford HELM Lite data69.9%
23Meta Llama 3 70b· Stanford HELM Lite data69.5%
24Nova Litev1· Stanford HELM Lite data69.3%
25Jamba 1.5 Large· Stanford HELM Lite data68.3%
26gemini-1.5-flash-002· Stanford HELM Lite data67.9%
27Phi-3 Medium 4K Instruct· Stanford HELM Lite data67.5%
28Claude Haiku 3.5· Stanford HELM Lite data67.1%
29GPT-4o mini· Stanford HELM Lite data66.8%
30Gemma 2 27B IT· Stanford HELM Lite data66.4%
31Claude 3 Haiku· Stanford HELM Lite data66.2%
32Phi-3 Small 8K Instruct· Stanford HELM Lite data65.9%
33Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data65.8%
34Claude 3 Sonnet· Stanford HELM Lite data65.2%
35Yi-34B· Stanford HELM Lite data65%
36Qwen1.5-72B· Stanford HELM Lite data64.7%
37Gemma 2 9B IT· Stanford HELM Lite data64.5%
38Claude 2.1· Stanford HELM Lite data64.3%
39DBRX-instruct· Stanford HELM Lite data64.3%
40DeepSeek LLM 67B Chat· Stanford HELM Lite data64.1%
41Nova Microv1· Stanford HELM Lite data64%
42Claude 2· Stanford HELM Lite data63.9%
43Mistral Large· Stanford HELM Lite data63.8%
44Claude Instant 1.2· Stanford HELM Lite data63.1%
45Qwen1.5-14B· Stanford HELM Lite data62.6%
46GPT-3.5 Turbo· Stanford HELM Lite data61.4%
47Open Mistral Nemo· Stanford HELM Lite data60.4%
48Meta Llama 3 8b· Stanford HELM Lite data60.2%
49Mistral Small· Stanford HELM Lite data59.3%
50Command R Plus· Stanford HELM Lite data59%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)