Skip to content

Benchmark

HELM Lite: GSM8K - EM

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
68
Scores recorded
69
Sources
1
IQ weight
1.78

Score distribution

71%
Mean
78.5%
Median
20.8%
Std dev
45.2%
10th percentile
93.2%
90th percentile
4.4%
Min
4.4%94.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.78 for chat models, measured over 68 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude 3.5 Sonnet· Stanford HELM Lite data94.9%
2Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data94.9%
3Gemini 2.0 Flash Experimental· Stanford HELM Lite data94.6%
4Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data94.2%
5DeepSeek V3· Stanford HELM Lite data94%
6Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data93.8%
7Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data93.6%
8GPT 4· Stanford HELM Lite data93.2%
9Claude 3 Opus· Stanford HELM Lite data92.4%
10Qwen2 72B Instruct· Stanford HELM Lite data92%
11Mistral Large2407· Stanford HELM Lite data91.2%
12Claude 3 Sonnet· Stanford HELM Lite data90.7%
13GPT 4o· Stanford HELM Lite data90.5%
14Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data90%
15Phi-3 Medium 4K Instruct· Stanford HELM Lite data87.8%
16Nova Prov1· Stanford HELM Lite data87%
17Jamba 1.5 Large· Stanford HELM Lite data84.6%
18GPT-4o mini· Stanford HELM Lite data84.3%
19gemini-1.5-pro-001· Stanford HELM Lite data83.6%
20text-unicorn· Stanford HELM Lite data83.1%
21Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data83%
22Nova Litev1· Stanford HELM Lite data82.9%
23GPT-4 Turbo· Stanford HELM Lite data82.4%
24Llama-3.2-11B-Vision-Instruct-Turbo· Stanford HELM Lite data82.3%
25gemini-1.5-pro-002· Stanford HELM Lite data81.7%
26Claude Haiku 3.5· Stanford HELM Lite data81.5%
27Qwen1.5-110B-Chat· Stanford HELM Lite data81.5%
28Gemma 2 27B IT· Stanford HELM Lite data81.2%
29Meta Llama 3 70b· Stanford HELM Lite data80.5%
30mixtral-8x22b· Stanford HELM Lite data80%
31Qwen1.5-72B· Stanford HELM Lite data79.9%
32Llama 3.1 8B Instruct Turbo· Stanford HELM Lite data79.8%
33DeepSeek LLM 67B Chat· Stanford HELM Lite data79.5%
34Nova Microv1· Stanford HELM Lite data79.4%
35gemini-1.5-flash-001· Stanford HELM Lite data78.5%
36Open Mistral Nemo· Stanford HELM Lite data78.2%
37Snowflake Arctic Instruct· Stanford HELM Lite data76.8%
38Gemma 2 9B IT· Stanford HELM Lite data76.2%
39Command R Plus· Stanford HELM Lite data73.8%
40Mistral Small· Stanford HELM Lite data73.4%
41Claude Instant 1.2· Stanford HELM Lite data72.1%
42Claude 3 Haiku· Stanford HELM Lite data69.9%
43Mistral Large· Stanford HELM Lite data69.4%
44Qwen1.5-14B· Stanford HELM Lite data69.3%
45Jamba 1.5 Mini· Stanford HELM Lite data69.1%
46DBRX-instruct· Stanford HELM Lite data67.1%
47Jamba Instruct· Stanford HELM Lite data67%
48GPT 4 1106 Preview· Stanford HELM Lite data66.8%
49Yi-34B· Stanford HELM Lite data64.8%
50Text Davinci 003· Stanford HELM Lite data61.5%

Showing 1–50 of 69 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)