Skip to content

Benchmark

HELM Lite: MedQA - EM

Science benchmarks measure how well models answer science questions. Scores are percentages.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
1.73

Score distribution

62.9%
Mean
65.6%
Median
14.3%
Std dev
44.5%
10th percentile
78.3%
90th percentile
22.9%
Min
22.9%85.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.73 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 4o· Stanford HELM Lite data85.7%
2Claude 3.5 Sonnet· Stanford HELM Lite data82.5%
3GPT 4 1106 Preview· Stanford HELM Lite data81.7%
4GPT 4· Stanford HELM Lite data81.5%
5Nova Prov1· Stanford HELM Lite data81.1%
6DeepSeek V3· Stanford HELM Lite data80.9%
7Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data80.5%
8GPT-4 Turbo· Stanford HELM Lite data78.3%
9Meta Llama 3 70b· Stanford HELM Lite data77.7%
10Claude 3 Opus· Stanford HELM Lite data77.5%
11Mistral Large2407· Stanford HELM Lite data77.5%
12gemini-1.5-pro-002· Stanford HELM Lite data77.1%
13Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data76.9%
14Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data76.9%
15Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data76.1%
16Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data75.3%
17GPT-4o mini· Stanford HELM Lite data74.8%
18Qwen2 72B Instruct· Stanford HELM Lite data74.6%
19Gemini 2.0 Flash Experimental· Stanford HELM Lite data73%
20Claude Haiku 3.5· Stanford HELM Lite data72.2%
21mixtral-8x22b· Stanford HELM Lite data70.4%
22Claude 3 Haiku· Stanford HELM Lite data70.2%
23Jamba 1.5 Large· Stanford HELM Lite data69.8%
24Nova Litev1· Stanford HELM Lite data69.6%
25Phi-3 Medium 4K Instruct· Stanford HELM Lite data69.6%
26DBRX-instruct· Stanford HELM Lite data69.4%
27gemini-1.5-pro-001· Stanford HELM Lite data69.2%
28Claude 3 Sonnet· Stanford HELM Lite data68.4%
29Gemma 2 27B IT· Stanford HELM Lite data68.4%
30text-unicorn· Stanford HELM Lite data68.4%
31gemini-1.5-flash-001· Stanford HELM Lite data68%
32Phi-3 Small 8K Instruct· Stanford HELM Lite data67.2%
33Qwen1.5-72B· Stanford HELM Lite data67%
34gemini-1.5-flash-002· Stanford HELM Lite data65.6%
35Yi-34B· Stanford HELM Lite data65.6%
36Claude 2· Stanford HELM Lite data65.2%
37Claude 2.1· Stanford HELM Lite data64.4%
38Qwen1.5-110B-Chat· Stanford HELM Lite data64%
39Jamba 1.5 Mini· Stanford HELM Lite data63.2%
40Gemma 2 9B IT· Stanford HELM Lite data63%
41DeepSeek LLM 67B Chat· Stanford HELM Lite data62.8%
42GPT-3.5 Turbo· Stanford HELM Lite data62.2%
43Llama 2 70b· Stanford HELM Lite data61.8%
44Mistral Small· Stanford HELM Lite data61.6%
45Nova Microv1· Stanford HELM Lite data60.8%
46Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data60%
47Open Mistral Nemo· Stanford HELM Lite data59%
48Meta Llama 3 8b· Stanford HELM Lite data58.1%
49Snowflake Arctic Instruct· Stanford HELM Lite data58.1%
50Command R Plus· Stanford HELM Lite data56.7%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)