Skip to content

Benchmark

HELM AIR-Bench: AIRBench 2024 - #39-40: Discrimination/Protected Characteristics Combinations

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages. Higher is better.

Models measured
70
Scores recorded
75
Sources
1
IQ weight
1.18

Score distribution

64%
Mean
61.8%
Median
14%
Std dev
45.8%
10th percentile
85.5%
90th percentile
41.9%
Min
41.9%90.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.18 for chat models, measured over 70 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5· Stanford HELM specialized suites data docs90.7%
2Claude Haiku 4.5· Stanford HELM specialized suites data docs90%
3Claude Sonnet 4.5· Stanford HELM specialized suites data docs88.9%
4Claude Sonnet 4· Stanford HELM specialized suites data docs88.2%
5GPT-5 nano· Stanford HELM specialized suites data docs88%
6GPT 5.1· Stanford HELM specialized suites data docs87.1%
7GPT-5 mini· Stanford HELM specialized suites data docs86.6%
8Claude Opus 4· Stanford HELM specialized suites data docs85.5%
9Claude 3.5 Sonnetsetting not named by the source· Stanford HELM specialized suites data docs84.8%
10OpenAI o3· Stanford HELM specialized suites data docs83.8%
11Qwen3 Next 80B A3Bthinking· Stanford HELM specialized suites data docs82.8%
12Claude 3.5 Sonnetsetting not named by the source· Stanford HELM specialized suites data docs81.4%
13Claude Sonnet 3.7· Stanford HELM specialized suites data docs80.3%
14GPT OSS 120B· Stanford HELM specialized suites data docs79.4%
15o1· Stanford HELM specialized suites data docs79.2%
16Claude 3 Opus· Stanford HELM specialized suites data docs78.3%
17Gemini 2.5 Pro Preview 03-25· Stanford HELM specialized suites data docs75.9%
18o4 Mini· Stanford HELM specialized suites data docs75.5%
19GPT OSS 20b· Stanford HELM specialized suites data docs74.8%
20Qwen3 235B A22B Instruct 2507· Stanford HELM specialized suites data docs74.6%
21gemini-2.5-flash-preview-04-17· Stanford HELM specialized suites data docs74.1%
22Claude 3 Sonnet· Stanford HELM specialized suites data docs73.9%
23gemini-1.5-pro-002· Stanford HELM specialized suites data docs72.8%
24o3 Mini· Stanford HELM specialized suites data docs72.5%
25Gemini 3 Pro Preview· Stanford HELM specialized suites data docs72.3%
26gemini-1.5-flash-002· Stanford HELM specialized suites data docs71.5%
27Claude 3 Haiku· Stanford HELM specialized suites data docs69%
28GPT-4.5 Preview· Stanford HELM specialized suites data docs67.5%
29Kimi K2 Instruct· Stanford HELM specialized suites data docs67.4%
30Palmyra X5· Stanford HELM specialized suites data docs67.4%
31gemini-2.0-pro-exp-02-05· Stanford HELM specialized suites data docs65.6%
32GPT-4 Turbo· Stanford HELM specialized suites data docs64.9%
33Gemini 2.5 Flash-Lite· Stanford HELM specialized suites data docs64.5%
34Grok 3 Mini· Stanford HELM specialized suites data docs63.2%
35GLM 4.5 Air· Stanford HELM specialized suites data docs62.7%
36O1 Mini· Stanford HELM specialized suites data docs62.5%
37Grok 3· Stanford HELM specialized suites data docs62.3%
38Gemini 2.0 Flash· Stanford HELM specialized suites data docs61.8%
39Granite 3.3 8b Instruct· Stanford HELM specialized suites data docs61.6%
40GPT 4.1· Stanford HELM specialized suites data docs61%
41Gemini 2.0 Flash-Lite Preview· Stanford HELM specialized suites data docs60%
42Granite 4.0 H Small· Stanford HELM specialized suites data docs60%
43GPT 4osetting not named by the source· Stanford HELM specialized suites data docs59.5%
44Llama 4 Maverick· Stanford HELM specialized suites data docs58.9%
45DeepSeek R1· Stanford HELM specialized suites data docs58.8%
46GPT-3.5 Turbosetting not named by the source· Stanford HELM specialized suites data docs57.5%
47GPT-3.5 Turbosetting not named by the source· Stanford HELM specialized suites data docs57.3%
48Qwen2 72B Instruct· Stanford HELM specialized suites data docs56.9%
49Llama3 8B Chat· Stanford HELM specialized suites data docs56.4%
50Grok 4· Stanford HELM specialized suites data docs56%

Showing 1–50 of 75 results.

Data credits: Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance