Skip to content

Benchmark

HELM Safety: BBQ - BBQ accuracy

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages. Higher is better.

Models measured
69
Scores recorded
74
Sources
1
IQ weight
1.95

Score distribution

90.2%
Mean
94.1%
Median
9.9%
Std dev
72.4%
10th percentile
97.7%
90th percentile
56.5%
Min
56.5%99.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.95 for chat models, measured over 69 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 4setting not named by the source· Stanford HELM specialized suites data docs99.3%
2Claude Sonnet 4.5· Stanford HELM specialized suites data docs98.9%
3GPT OSS 120B· Stanford HELM specialized suites data docs98.5%
4Gemini 3 Pro Preview· Stanford HELM specialized suites data docs98.4%
5Claude Sonnet 4setting not named by the source· Stanford HELM specialized suites data docs97.9%
6OpenAI o3· Stanford HELM specialized suites data docs97.9%
7GLM 4.5 Air· Stanford HELM specialized suites data docs97.8%
8gemini-2.5-flash-preview-04-17· Stanford HELM specialized suites data docs97.7%
9GPT-5 nano· Stanford HELM specialized suites data docs97.6%
10o1· Stanford HELM specialized suites data docs97.3%
11Claude Opus 4thinking, 10k· Stanford HELM specialized suites data docs97.1%
12O1 Mini· Stanford HELM specialized suites data docs96.9%
13GPT 5· Stanford HELM specialized suites data docs96.8%
14DeepSeek V3· Stanford HELM specialized suites data docs96.7%
15GPT OSS 20b· Stanford HELM specialized suites data docs96.7%
16Grok 3 Mini· Stanford HELM specialized suites data docs96.7%
17Claude Sonnet 4thinking, 10k· Stanford HELM specialized suites data docs96.6%
18DeepSeek R1setting not named by the source· Stanford HELM specialized suites data docs96.6%
19DeepSeek R1setting not named by the source· Stanford HELM specialized suites data docs96.5%
20Gemini 2.5 Pro Preview 03-25· Stanford HELM specialized suites data docs96.4%
21GPT-5 mini· Stanford HELM specialized suites data docs96.3%
22Qwen3 235B A22B Instruct 2507· Stanford HELM specialized suites data docs96.2%
23gemini-2.0-pro-exp-02-05· Stanford HELM specialized suites data docs95.6%
24Gemini 2.0 Flash· Stanford HELM specialized suites data docs95.4%
25Meta Llama 3.1 70B Instruct Turbo· Stanford HELM specialized suites data docs95.4%
26Qwen2.5 72B Instruct Turbo· Stanford HELM specialized suites data docs95.4%
27GPT 4o· Stanford HELM specialized suites data docs95.1%
28Qwen2 72B Instruct· Stanford HELM specialized suites data docs95.1%
29Claude 3.5 Sonnet· Stanford HELM specialized suites data docs94.9%
30Gemini 2.5 Flash-Lite· Stanford HELM specialized suites data docs94.9%
31Kimi K2 Instruct· Stanford HELM specialized suites data docs94.9%
32Palmyra X5· Stanford HELM specialized suites data docs94.8%
33gemini-1.5-flash-001· Stanford HELM specialized suites data docs94.7%
34gemini-1.5-pro-001· Stanford HELM specialized suites data docs94.5%
35Llama 3.1 405B Instruct Turbo· Stanford HELM specialized suites data docs94.5%
36GPT-4 Turbo· Stanford HELM specialized suites data docs94.1%
37o3 Mini· Stanford HELM specialized suites data docs94.1%
38Claude 3 Opus· Stanford HELM specialized suites data docs94%
39o4 Mini· Stanford HELM specialized suites data docs94%
40Grok 4· Stanford HELM specialized suites data docs93.7%
41Grok 3· Stanford HELM specialized suites data docs93.6%
42Mistral Small· Stanford HELM specialized suites data docs93.3%
43Llama 4 Maverick· Stanford HELM specialized suites data docs93%
44Claude Haiku 4.5· Stanford HELM specialized suites data docs92.8%
45GPT 4.1· Stanford HELM specialized suites data docs92.6%
46Granite 4.0 H Small· Stanford HELM specialized suites data docs92.5%
47Claude Sonnet 3.7· Stanford HELM specialized suites data docs92.1%
48GPT-4.1 mini· Stanford HELM specialized suites data docs92.1%
49Gemini 2.0 Flash-Lite Preview· Stanford HELM specialized suites data docs92%
50GPT-4.5 Preview· Stanford HELM specialized suites data docs92%

Showing 1–50 of 74 results.

Data credits: Stanford HELM (Stanford CRFM); scenario scores from Capabilities, Safety, Long Context, MedHELM, AIR-Bench, MMLU and Finance