Skip to content

Benchmark

HELM AIR-Bench: AIRBench 2024 - #18: Erotic

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages. Higher is better.

Models measured
70
Scores recorded
75
Sources
1
IQ weight
0.60

Score distribution

39.1%
Mean
30%
Median
21.8%
Std dev
18.3%
10th percentile
75%
90th percentile
11.7%
Min
11.7%95%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.60 for chat models, measured over 70 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Mistral (7B) Instruct v0.3· Stanford HELM specialized suites data docs25%
52Mistral Large2407· Stanford HELM specialized suites data docs25%
53GPT-4.1 mini· Stanford HELM specialized suites data docs23.3%
54GPT-4.5 Preview· Stanford HELM specialized suites data docs23.3%
55Grok 3 Mini· Stanford HELM specialized suites data docs23.3%
56DeepSeek R1· Stanford HELM specialized suites data docs21.7%
57DeepSeek V3· Stanford HELM specialized suites data docs21.7%
58Gemini 2.5 Flash-Lite· Stanford HELM specialized suites data docs21.7%
59Gemini 3 Pro Preview· Stanford HELM specialized suites data docs21.7%
60gemini-1.5-flash-002· Stanford HELM specialized suites data docs21.7%
61GLM 4.5 Air· Stanford HELM specialized suites data docs21.7%
62Mistral Small· Stanford HELM specialized suites data docs21.7%
63Gemini 2.5 Pro Preview 03-25· Stanford HELM specialized suites data docs20%
64GPT-3.5 Turbosetting not named by the source· Stanford HELM specialized suites data docs20%
65GPT-4o mini· Stanford HELM specialized suites data docs20%
66o3 Mini· Stanford HELM specialized suites data docs20%
67gemini-1.5-pro-002· Stanford HELM specialized suites data docs18.3%
68gemini-2.5-flash-preview-04-17· Stanford HELM specialized suites data docs18.3%
69GPT 4.1· Stanford HELM specialized suites data docs18.3%
70Command R Plus· Stanford HELM specialized suites data docs16.7%
71Qwen1.5-72B-Chat· Stanford HELM specialized suites data docs16.7%
72Command R+· Stanford HELM specialized suites data docs15%
73Grok 3· Stanford HELM specialized suites data docs15%
74o1· Stanford HELM specialized suites data docs15%
75Grok 4· Stanford HELM specialized suites data docs11.7%

Showing 51–75 of 75 results.

Data credits: Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance