Skip to content

Benchmark

HELM Lite: Mean win rate

Scores are percentages. Higher is better.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
1.33

Score distribution

52.2%
Mean
48.9%
Median
22.8%
Std dev
26.3%
10th percentile
84.2%
90th percentile
5.2%
Min
5.2%93.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.33 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 4o· Stanford HELM Lite data93.8%
2DeepSeek V3· Stanford HELM Lite data90.8%
3Claude 3.5 Sonnet· Stanford HELM Lite data88.5%
4Nova Prov1· Stanford HELM Lite data88.5%
5GPT 4· Stanford HELM Lite data86.7%
6GPT-4 Turbo· Stanford HELM Lite data86.4%
7Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data85.4%
8gemini-1.5-pro-002· Stanford HELM Lite data84.2%
9Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data81.9%
10Gemini 2.0 Flash Experimental· Stanford HELM Lite data81.3%
11Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data81.2%
12Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data80.8%
13Meta Llama 3 70b· Stanford HELM Lite data79.3%
14Qwen2 72B Instruct· Stanford HELM Lite data77%
15Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data74.5%
16Mistral Large2407· Stanford HELM Lite data74.4%
17gemini-1.5-pro-001· Stanford HELM Lite data73.9%
18Nova Litev1· Stanford HELM Lite data70.8%
19mixtral-8x22b· Stanford HELM Lite data70.5%
20GPT-4o mini· Stanford HELM Lite data70.1%
21GPT 4 1106 Preview· Stanford HELM Lite data69.8%
22Claude 3 Opus· Stanford HELM Lite data68.3%
23Gemma 2 27B IT· Stanford HELM Lite data67.5%
24gemini-1.5-flash-001· Stanford HELM Lite data66.7%
25text-unicorn· Stanford HELM Lite data64.4%
26Jamba 1.5 Large· Stanford HELM Lite data63.7%
27Qwen1.5-72B· Stanford HELM Lite data60.8%
28gemini-1.5-flash-002· Stanford HELM Lite data57.3%
29Yi-34B· Stanford HELM Lite data57%
30Gemma 2 9B IT· Stanford HELM Lite data56.2%
31Qwen1.5-110B-Chat· Stanford HELM Lite data55%
32Claude Haiku 3.5· Stanford HELM Lite data53.1%
33Nova Microv1· Stanford HELM Lite data52.4%
34Phi-3 Medium 4K Instruct· Stanford HELM Lite data50.9%
35Claude 2· Stanford HELM Lite data48.9%
36DeepSeek LLM 67B Chat· Stanford HELM Lite data48.8%
37Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data48.8%
38Llama 2 70b· Stanford HELM Lite data48.2%
39Phi-3 Small 8K Instruct· Stanford HELM Lite data47.3%
40Command R Plus· Stanford HELM Lite data44.1%
41Text Davinci 003· Stanford HELM Lite data43.9%
42Claude 2.1· Stanford HELM Lite data43.7%
43Qwen1.5-14B· Stanford HELM Lite data42.5%
44Jamba 1.5 Mini· Stanford HELM Lite data41.4%
45Claude Instant 1.2· Stanford HELM Lite data39.9%
46Meta Llama 3 8b· Stanford HELM Lite data38.7%
47Claude 3 Sonnet· Stanford HELM Lite data37.7%
48GPT-3.5 Turbo· Stanford HELM Lite data35.8%
49Snowflake Arctic Instruct· Stanford HELM Lite data33.8%
50gemma-7b· Stanford HELM Lite data33.6%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)