Skip to content

Benchmark

HELM Lite: MATH - Equivalent (CoT)

Math benchmarks measure how well models solve maths problems. Scores are percentages. Higher is better.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
1.57

Score distribution

59.2%
Mean
67.4%
Median
24.7%
Std dev
23.6%
10th percentile
83.5%
90th percentile
2.9%
Min
2.9%92%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.57 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1gemini-1.5-pro-002· Stanford HELM Lite data92%
2DeepSeek V3· Stanford HELM Lite data91.2%
3gemini-1.5-flash-002· Stanford HELM Lite data90.8%
4Gemini 2.0 Flash Experimental· Stanford HELM Lite data90.1%
5Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data88.4%
6Claude Haiku 3.5· Stanford HELM Lite data87.2%
7GPT 4 1106 Preview· Stanford HELM Lite data85.7%
8Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data83.5%
9GPT-4 Turbo· Stanford HELM Lite data83.3%
10GPT 4o· Stanford HELM Lite data82.9%
11Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data82.7%
12gemini-1.5-pro-001· Stanford HELM Lite data82.5%
13Nova Prov1· Stanford HELM Lite data82.1%
14Claude 3.5 Sonnet· Stanford HELM Lite data81.3%
15Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data80.8%
16GPT 4· Stanford HELM Lite data80.2%
17GPT-4o mini· Stanford HELM Lite data80.2%
18Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data79.1%
19Qwen2 72B Instruct· Stanford HELM Lite data79%
20Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data78.3%
21Nova Litev1· Stanford HELM Lite data77.9%
22Claude 3 Opus· Stanford HELM Lite data76%
23Nova Microv1· Stanford HELM Lite data76%
24gemini-1.5-flash-001· Stanford HELM Lite data75.3%
25Mistral Large· Stanford HELM Lite data75%
26Gemma 2 27B IT· Stanford HELM Lite data74.6%
27Llama-3.2-11B-Vision-Instruct-Turbo· Stanford HELM Lite data73.9%
28Gemma 2 9B IT· Stanford HELM Lite data72.4%
29Phi-3 Small 8K Instruct· Stanford HELM Lite data70.3%
30Llama 3.1 8B Instruct Turbo· Stanford HELM Lite data70.3%
31Jamba 1.5 Large· Stanford HELM Lite data69.2%
32Qwen1.5-14B· Stanford HELM Lite data68.6%
33Qwen1.5-72B· Stanford HELM Lite data68.3%
34Mistral Large2407· Stanford HELM Lite data67.7%
35text-unicorn· Stanford HELM Lite data67.4%
36Open Mistral Nemo· Stanford HELM Lite data66.8%
37GPT-3.5 Turbo· Stanford HELM Lite data66.7%
38Meta Llama 3 70b· Stanford HELM Lite data66.3%
39mixtral-8x22b· Stanford HELM Lite data65.6%
40Claude 2.1· Stanford HELM Lite data63.2%
41Mistral Small· Stanford HELM Lite data62.1%
42DeepSeek LLM 67B Chat· Stanford HELM Lite data61.5%
43Phi-3 Medium 4K Instruct· Stanford HELM Lite data61.1%
44Claude 2· Stanford HELM Lite data60.3%
45Qwen1.5-110B-Chat· Stanford HELM Lite data56.8%
46Qwen1.5-7B· Stanford HELM Lite data56.1%
47Snowflake Arctic Instruct· Stanford HELM Lite data51.9%
48gemma-7b· Stanford HELM Lite data50%
49Claude Instant 1.2· Stanford HELM Lite data49.9%
50Text Davinci 003· Stanford HELM Lite data44.9%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)