Skip to content

Benchmark

HELM Capabilities: MMLU-Pro - COT correct

Scores are percentages. Higher is better.

Models measured
57
Scores recorded
59
Sources
1
IQ weight
1.62

Score distribution

70.1%
Mean
74%
Median
15%
Std dev
51.1%
10th percentile
85.9%
90th percentile
27.7%
Min
27.7%90.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.62 for chat models, measured over 57 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 3 Pro Preview· Stanford HELM specialized suites data docs90.3%
2Claude Opus 4thinking, 10k· Stanford HELM specialized suites data docs87.5%
3Claude Sonnet 4.5· Stanford HELM specialized suites data docs86.9%
4Gemini 2.5 Pro Preview 03-25· Stanford HELM specialized suites data docs86.3%
5GPT 5· Stanford HELM specialized suites data docs86.3%
6Claude Opus 4setting not named by the source· Stanford HELM specialized suites data docs85.9%
7OpenAI o3· Stanford HELM specialized suites data docs85.9%
8Grok 4· Stanford HELM specialized suites data docs85.1%
9Qwen3 235B A22B Instruct 2507· Stanford HELM specialized suites data docs84.4%
10Claude Sonnet 4setting not named by the source· Stanford HELM specialized suites data docs84.3%
11Claude Sonnet 4thinking, 10k· Stanford HELM specialized suites data docs84.3%
12GPT-5 mini· Stanford HELM specialized suites data docs83.5%
13o4 Mini· Stanford HELM specialized suites data docs82%
14Kimi K2 Instruct· Stanford HELM specialized suites data docs81.9%
15GPT 4.1· Stanford HELM specialized suites data docs81.1%
16Llama 4 Maverick· Stanford HELM specialized suites data docs81%
17Palmyra X5· Stanford HELM specialized suites data docs80.4%
18Grok 3 Mini· Stanford HELM specialized suites data docs79.9%
19GPT OSS 120B· Stanford HELM specialized suites data docs79.5%
20DeepSeek R1· Stanford HELM specialized suites data docs79.3%
21Grok 3· Stanford HELM specialized suites data docs78.8%
22Qwen3 Next 80B A3Bthinking· Stanford HELM specialized suites data docs78.6%
23Claude Sonnet 3.7· Stanford HELM specialized suites data docs78.4%
24GPT-4.1 mini· Stanford HELM specialized suites data docs78.3%
25GPT-5 nano· Stanford HELM specialized suites data docs77.8%
26Claude 3.5 Sonnet· Stanford HELM specialized suites data docs77.7%
27Claude Haiku 4.5· Stanford HELM specialized suites data docs77.7%
28GLM 4.5 Air· Stanford HELM specialized suites data docs76.2%
29Llama 4 Scout· Stanford HELM specialized suites data docs74.2%
30GPT OSS 20b· Stanford HELM specialized suites data docs74%
31Gemini 2.0 Flash· Stanford HELM specialized suites data docs73.7%
32gemini-1.5-pro-002· Stanford HELM specialized suites data docs73.7%
33Nova Premier 1.0· Stanford HELM specialized suites data docs72.6%
34DeepSeek V3· Stanford HELM specialized suites data docs72.3%
35Llama 3.1 405B Instruct Turbo· Stanford HELM specialized suites data docs72.3%
36Gemini 2.0 Flash-Lite Preview· Stanford HELM specialized suites data docs72%
37GPT 4o· Stanford HELM specialized suites data docs71.3%
38gemini-1.5-flash-002· Stanford HELM specialized suites data docs67.8%
39Nova Prov1· Stanford HELM specialized suites data docs67.3%
40Meta Llama 3.1 70B Instruct Turbo· Stanford HELM specialized suites data docs65.3%
41gemini-2.5-flash-preview-04-17· Stanford HELM specialized suites data docs63.9%
42Qwen2.5 72B Instruct Turbo· Stanford HELM specialized suites data docs63.1%
43Mistral Small· Stanford HELM specialized suites data docs61%
44Claude Haiku 3.5· Stanford HELM specialized suites data docs60.5%
45GPT-4o mini· Stanford HELM specialized suites data docs60.3%
46Nova Litev1· Stanford HELM specialized suites data docs60%
47Mistral Large2411· Stanford HELM specialized suites data docs59.9%
48GPT 5.1· Stanford HELM specialized suites data docs57.9%
49Granite 4.0 H Small· Stanford HELM specialized suites data docs56.9%
50GPT-4.1 nano· Stanford HELM specialized suites data docs55%

Showing 1–50 of 59 results.

Data credits: Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance