Skip to content

Benchmark

HELM Lite: LegalBench - EM

Scores are percentages.

Models measured
69
Scores recorded
70
Sources
1
IQ weight
1.99

Score distribution

60.2%
Mean
63.2%
Median
11.1%
Std dev
42.6%
10th percentile
72.5%
90th percentile
33.1%
Min
33.1%75.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.99 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1gemini-1.5-pro-001· Stanford HELM Lite data75.7%
2gemini-1.5-pro-002· Stanford HELM Lite data74.7%
3Qwen2.5 72B Instruct Turbo· Stanford HELM Lite data74%
4Nova Prov1· Stanford HELM Lite data73.6%
5GPT 4o· Stanford HELM Lite data73.3%
6Meta Llama 3 70b· Stanford HELM Lite data73.3%
7GPT-4 Turbo· Stanford HELM Lite data72.7%
8Llama 3.3 70B Instruct Turbo· Stanford HELM Lite data72.5%
9DeepSeek V3· Stanford HELM Lite data71.8%
10GPT 4· Stanford HELM Lite data71.3%
11Qwen2 72B Instruct· Stanford HELM Lite data71.2%
12mixtral-8x22b· Stanford HELM Lite data70.8%
13Llama 3.1 405B Instruct Turbo· Stanford HELM Lite data70.7%
14Claude 3.5 Sonnet· Stanford HELM Lite data70.7%
15Gemma 2 27B IT· Stanford HELM Lite data70%
16Qwen1.5-72B· Stanford HELM Lite data69.4%
17Meta Llama 3.1 70B Instruct Turbo· Stanford HELM Lite data68.7%
18Llama-3.2-90B-Vision-Instruct-Turbo· Stanford HELM Lite data68%
19text-unicorn· Stanford HELM Lite data67.7%
20Jamba 1.5 Large· Stanford HELM Lite data67.5%
21Gemini 2.0 Flash Experimental· Stanford HELM Lite data67.4%
22Llama 2 70b· Stanford HELM Lite data67.3%
23Command R Plus· Stanford HELM Lite data67.2%
24gemini-1.5-flash-002· Stanford HELM Lite data67%
25Claude 3 Opus· Stanford HELM Lite data66.2%
26gemini-1.5-flash-001· Stanford HELM Lite data66.1%
27Nova Litev1· Stanford HELM Lite data65.9%
28GPT-4o mini· Stanford HELM Lite data65.3%
29Mistral Large2407· Stanford HELM Lite data64.6%
30Claude 2· Stanford HELM Lite data64.3%
31Claude 2.1· Stanford HELM Lite data64.3%
32Gemma 2 9B IT· Stanford HELM Lite data63.9%
33Meta Llama 3 8b· Stanford HELM Lite data63.7%
34DeepSeek LLM 67B Chat· Stanford HELM Lite data63.7%
35Qwen2.5 7B Instruct Turbo· Stanford HELM Lite data63.2%
36Claude Haiku 3.5· Stanford HELM Lite data63.1%
37GPT 4 1106 Preview· Stanford HELM Lite data62.6%
38Qwen1.5-110B-Chat· Stanford HELM Lite data62.4%
39Text Davinci 003· Stanford HELM Lite data62.2%
40Yi-34B· Stanford HELM Lite data61.8%
41Nova Microv1· Stanford HELM Lite data61.5%
42Qwen1.5-14B· Stanford HELM Lite data59.3%
43Phi-3 Medium 4K Instruct· Stanford HELM Lite data59.3%
44Llama 2 13b· Stanford HELM Lite data59.1%
45Snowflake Arctic Instruct· Stanford HELM Lite data58.8%
46Claude Instant 1.2· Stanford HELM Lite data58.6%
47Phi-3 Small 8K Instruct· Stanford HELM Lite data58.4%
48gemma-7b· Stanford HELM Lite data58.1%
49Mistral 7B v0.1· Stanford HELM Lite data58%
50Text Davinci 002· Stanford HELM Lite data58%

Showing 1–50 of 70 results.

Data credits: Stanford HELM (Stanford CRFM), Apache-2.0 (licence)