Skip to content

Benchmark

HELM AIR-Bench: AIRBench 2024 - Operational Misuses

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages. Higher is better.

Models measured
70
Scores recorded
75
Sources
1
IQ weight
0.80

Score distribution

37.1%
Mean
34.5%
Median
15.6%
Std dev
17.1%
10th percentile
59%
90th percentile
10.9%
Min
10.9%75.4%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.80 for chat models, measured over 70 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Haiku 4.5· Stanford HELM specialized suites data docs75.4%
2Claude 3.5 Sonnetsetting not named by the source· Stanford HELM specialized suites data docs65.5%
3GPT OSS 120B· Stanford HELM specialized suites data docs65.4%
4Claude Sonnet 4.5· Stanford HELM specialized suites data docs62.6%
5Qwen3 Next 80B A3Bthinking· Stanford HELM specialized suites data docs60.8%
6GPT-5 nano· Stanford HELM specialized suites data docs60.5%
7OpenAI o3· Stanford HELM specialized suites data docs60.3%
8Claude Sonnet 4· Stanford HELM specialized suites data docs59%
9Claude Opus 4· Stanford HELM specialized suites data docs57.4%
10gemini-1.5-flash-002· Stanford HELM specialized suites data docs57.2%
11GPT OSS 20b· Stanford HELM specialized suites data docs56.5%
12Claude 3 Opus· Stanford HELM specialized suites data docs56%
13GPT 5· Stanford HELM specialized suites data docs55.5%
14Claude 3 Sonnet· Stanford HELM specialized suites data docs55.3%
15Claude 3.5 Sonnetsetting not named by the source· Stanford HELM specialized suites data docs55.1%
16GPT 5.1· Stanford HELM specialized suites data docs53.8%
17GPT-3.5 Turbosetting not named by the source· Stanford HELM specialized suites data docs53.1%
18Claude Sonnet 3.7· Stanford HELM specialized suites data docs51%
19GPT-5 mini· Stanford HELM specialized suites data docs50%
20GPT-3.5 Turbosetting not named by the source· Stanford HELM specialized suites data docs49.9%
21GPT 4· Stanford HELM specialized suites data docs49.4%
22gemini-1.5-pro-002· Stanford HELM specialized suites data docs47.6%
23o1· Stanford HELM specialized suites data docs46%
24Claude 3 Haiku· Stanford HELM specialized suites data docs45.5%
25Palmyra X5· Stanford HELM specialized suites data docs44.7%
26gemini-2.5-flash-preview-04-17· Stanford HELM specialized suites data docs43.2%
27Gemini 2.5 Pro Preview 03-25· Stanford HELM specialized suites data docs42.6%
28Gemini 3 Pro Preview· Stanford HELM specialized suites data docs41.8%
29gemini-2.0-pro-exp-02-05· Stanford HELM specialized suites data docs40.4%
30GPT-3.5 Turbosetting not named by the source· Stanford HELM specialized suites data docs40.1%
31GPT-4 Turbo· Stanford HELM specialized suites data docs40.1%
32Granite 4.0 H Small· Stanford HELM specialized suites data docs39.5%
33o3 Mini· Stanford HELM specialized suites data docs39.1%
34o4 Mini· Stanford HELM specialized suites data docs38.2%
35Granite 3.3 8b Instruct· Stanford HELM specialized suites data docs36.4%
36Gemini 2.0 Flash· Stanford HELM specialized suites data docs36.3%
37Gemini 2.5 Flash-Lite· Stanford HELM specialized suites data docs35.5%
38DeepSeek LLM 67B Chat· Stanford HELM specialized suites data docs34.5%
39Yi 34B Chat· Stanford HELM specialized suites data docs34.2%
40Gemini 2.0 Flash-Lite Preview· Stanford HELM specialized suites data docs33.6%
41Qwen3 235B A22B Instruct 2507· Stanford HELM specialized suites data docs33.1%
42Llama 4 Maverick· Stanford HELM specialized suites data docs32.8%
43Llama3 8B Chat· Stanford HELM specialized suites data docs32.1%
44GPT 4osetting not named by the source· Stanford HELM specialized suites data docs31.7%
45Llama 3.1 405B Instruct Turbo· Stanford HELM specialized suites data docs31.3%
46Llama 3.1 8B Instruct Turbo· Stanford HELM specialized suites data docs31%
47GPT-4.5 Preview· Stanford HELM specialized suites data docs30.9%
48Mixtral 8x22B Instruct v0.1· Stanford HELM specialized suites data docs28.8%
49O1 Mini· Stanford HELM specialized suites data docs28.3%
50Qwen1.5-72B-Chat· Stanford HELM specialized suites data docs27.4%

Showing 1–50 of 75 results.

Data credits: Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance