Skip to content

Benchmark

HELM MMLU: Formal Logic - EM

Scores are percentages. Higher is better.

Models measured
64
Scores recorded
68
Sources
1
IQ weight
1.45

Score distribution

55.7%
Mean
57.1%
Median
11.9%
Std dev
39.7%
10th percentile
69.8%
90th percentile
27%
Min
27%78.6%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.45 for chat models, measured over 64 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude 3.5 Sonnetsetting not named by the source· Stanford HELM specialized suites data docs78.6%
2DeepSeek V3· Stanford HELM specialized suites data docs77%
3gemini-1.5-pro-002· Stanford HELM specialized suites data docs75.4%
4Qwen2.5 72B Instruct Turbo· Stanford HELM specialized suites data docs73%
5Nova Prov1· Stanford HELM specialized suites data docs71.4%
6gemini-1.5-pro-001· Stanford HELM specialized suites data docs70.6%
7GPT-4 Turbo· Stanford HELM specialized suites data docs70.6%
8Claude 3 Opus· Stanford HELM specialized suites data docs69.8%
9Claude 3.5 Sonnetsetting not named by the source· Stanford HELM specialized suites data docs69.8%
10Llama 3.1 405B Instruct Turbo· Stanford HELM specialized suites data docs69.8%
11GPT 4osetting not named by the source· Stanford HELM specialized suites data docs68.3%
12Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM specialized suites data docs68.3%
13GPT 4osetting not named by the source· Stanford HELM specialized suites data docs67.5%
14Llama 3.3 70B Instruct Turbo· Stanford HELM specialized suites data docs67.5%
15Meta Llama 3.1 70B Instruct Turbo· Stanford HELM specialized suites data docs67.5%
16Qwen2 72B Instruct· Stanford HELM specialized suites data docs66.7%
17text-unicorn· Stanford HELM specialized suites data docs65.9%
18GPT 4 1106 Preview· Stanford HELM specialized suites data docs65.1%
19Meta Llama 3 70b· Stanford HELM specialized suites data docs65.1%
20GPT 4· Stanford HELM specialized suites data docs64.3%
21Nova Litev1· Stanford HELM specialized suites data docs64.3%
22Qwen1.5-110B-Chat· Stanford HELM specialized suites data docs64.3%
23gemini-1.5-flash-001· Stanford HELM specialized suites data docs62.7%
24mixtral-8x22b· Stanford HELM specialized suites data docs62.7%
25Claude Haiku 3.5· Stanford HELM specialized suites data docs61.9%
26Jamba 1.5 Large· Stanford HELM specialized suites data docs61.9%
27gemini-1.5-flash-002· Stanford HELM specialized suites data docs59.5%
28Phi-3 Small 8K Instruct· Stanford HELM specialized suites data docs59.5%
29Phi-3 Medium 4K Instruct· Stanford HELM specialized suites data docs58.7%
30Qwen2.5 7B Instruct Turbo· Stanford HELM specialized suites data docs58.7%
31Claude 3 Haiku· Stanford HELM specialized suites data docs57.9%
32Claude 3 Sonnet· Stanford HELM specialized suites data docs57.9%
33Mistral Large2407· Stanford HELM specialized suites data docs57.9%
34Gemini 2.0 Flash Experimental· Stanford HELM specialized suites data docs57.1%
35DBRX-instruct· Stanford HELM specialized suites data docs56.3%
36GPT-4o mini· Stanford HELM specialized suites data docs55.6%
37Qwen1.5-72B· Stanford HELM specialized suites data docs55.6%
38DeepSeek LLM 67B Chat· Stanford HELM specialized suites data docs54.8%
39Yi-34B· Stanford HELM specialized suites data docs54.8%
40Mistral Largesetting not named by the source· Stanford HELM specialized suites data docs53.2%
41gemma2-27b· Stanford HELM specialized suites data docs51.6%
42Mistral Small· Stanford HELM specialized suites data docs51.6%
43gemma-7b· Stanford HELM specialized suites data docs50.8%
44Nova Microv1· Stanford HELM specialized suites data docs50.8%
45Claude 2.1· Stanford HELM specialized suites data docs50%
46gemma-2-9b· Stanford HELM specialized suites data docs49.2%
47Qwen1.5-14B· Stanford HELM specialized suites data docs49.2%
48Command R Plus· Stanford HELM specialized suites data docs48.4%
49Llama 2 70b· Stanford HELM specialized suites data docs46.8%
50Meta Llama 3 8b· Stanford HELM specialized suites data docs46.8%

Showing 1–50 of 68 results.

Data credits: Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance