Skip to content

Benchmark

AHELM: Voice Jailbreak Attacks Against GPT-4o - Refusal rate for safety

Scores are percentages.

Models measured
12
Scores recorded
36
Sources
1
IQ weight
0.32

Score distribution

57.1%
Mean
51.1%
Median
31.8%
Std dev
17.3%
10th percentile
96.7%
90th percentile
14%
Min
14%98.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.32 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT-4o Audio Previewsubset: textjailbreak· AHELM (HELM Audio) data98.7%
2GPT-4o Audio Preview· AHELM (HELM Audio) data97.8%
3GPT-4o mini Audio Previewsubset: textjailbreak· AHELM (HELM Audio) data97.3%
4Gemini 2.0 Flashsubset: baseline· AHELM (HELM Audio) data96.7%
5GPT-4o mini Audio Preview· AHELM (HELM Audio) data96.7%
6Gemini 2.0 Flash Experimentalsubset: baseline· AHELM (HELM Audio) data93.3%
7Gemini 2.5 Pro Preview 05-06subset: baseline· AHELM (HELM Audio) data93.3%
8gemini-1.5-pro-001subset: baseline· AHELM (HELM Audio) data93.3%
9GPT-4o Audio Previewsubset: baseline· AHELM (HELM Audio) data93.3%
10GPT-4o mini Audio Previewsubset: baseline· AHELM (HELM Audio) data93.3%
11gemini-1.5-flash-002subset: baseline· AHELM (HELM Audio) data90%
12Gemini 2.5 Flash Preview 05-20 (no think)subset: baseline· AHELM (HELM Audio) data86.7%
13gemini-1.5-pro-002subset: baseline· AHELM (HELM Audio) data86.7%
14gemini-1.5-flash-001subset: baseline· AHELM (HELM Audio) data83.3%
15Gemini 2.0 Flash Litesubset: baseline· AHELM (HELM Audio) data76.7%
16Qwen2.5-Omni 7Bsubset: baseline· AHELM (HELM Audio) data70%
17Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data53.3%
18gemini-1.5-pro-001· AHELM (HELM Audio) data51.1%
19Qwen2.5-Omni 7B· AHELM (HELM Audio) data51.1%
20Qwen2.5-Omni 7Bsubset: textjailbreak· AHELM (HELM Audio) data47.3%
21Gemini 2.5 Pro Preview 05-06subset: textjailbreak· AHELM (HELM Audio) data45.3%
22gemini-1.5-pro-001subset: textjailbreak· AHELM (HELM Audio) data42.7%
23gemini-1.5-flash-001· AHELM (HELM Audio) data31.7%
24Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data31.1%
25Gemini 2.0 Flash· AHELM (HELM Audio) data30.6%
26Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data28.9%
27gemini-1.5-flash-002· AHELM (HELM Audio) data26.7%
28gemini-1.5-pro-002· AHELM (HELM Audio) data26.1%
29Gemini 2.0 Flash Lite· AHELM (HELM Audio) data25%
30gemini-1.5-flash-001subset: textjailbreak· AHELM (HELM Audio) data21.3%
31Gemini 2.0 Flash Experimentalsubset: textjailbreak· AHELM (HELM Audio) data18.7%
32Gemini 2.0 Flashsubset: textjailbreak· AHELM (HELM Audio) data17.3%
33Gemini 2.5 Flash Preview 05-20 (no think)subset: textjailbreak· AHELM (HELM Audio) data17.3%
34Gemini 2.0 Flash Litesubset: textjailbreak· AHELM (HELM Audio) data14.7%
35gemini-1.5-flash-002subset: textjailbreak· AHELM (HELM Audio) data14%
36gemini-1.5-pro-002subset: textjailbreak· AHELM (HELM Audio) data14%