Skip to content

Benchmark

AHELM: Air-Bench Chat (reasoning subsets) - GPT-4o Judge Critique

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages.

Models measured
12
Scores recorded
36
Sources
1
IQ weight
1.17

Score distribution

3.1%
Mean
3.2%
Median
0.476%
Std dev
2.5%
10th percentile
3.7%
90th percentile
2.1%
Min
2.1%3.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.17 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Pro Preview 05-06subject: speech· AHELM (HELM Audio) data3.9%
2Gemini 2.5 Flash Preview 05-20 (no think)subject: speech· AHELM (HELM Audio) data3.9%
3Gemini 2.0 Flash Experimentalsubject: speech· AHELM (HELM Audio) data3.7%
4Gemini 2.0 Flashsubject: speech· AHELM (HELM Audio) data3.7%
5Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data3.6%
6gemini-1.5-flash-002subject: speech· AHELM (HELM Audio) data3.6%
7gemini-1.5-pro-002subject: speech· AHELM (HELM Audio) data3.5%
8GPT-4o Audio Previewsubject: speech· AHELM (HELM Audio) data3.5%
9Gemini 2.0 Flash Litesubject: speech· AHELM (HELM Audio) data3.5%
10Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data3.5%
11gemini-1.5-flash-001subject: speech· AHELM (HELM Audio) data3.4%
12gemini-1.5-pro-001subject: speech· AHELM (HELM Audio) data3.4%
13Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data3.3%
14Qwen2.5-Omni 7Bsubject: speech· AHELM (HELM Audio) data3.3%
15Gemini 2.0 Flash· AHELM (HELM Audio) data3.3%
16GPT-4o mini Audio Previewsubject: speech· AHELM (HELM Audio) data3.3%
17gemini-1.5-pro-002· AHELM (HELM Audio) data3.2%
18gemini-1.5-flash-002· AHELM (HELM Audio) data3.2%
19Gemini 2.0 Flash Lite· AHELM (HELM Audio) data3.2%
20GPT-4o Audio Preview· AHELM (HELM Audio) data3.2%
21gemini-1.5-flash-001· AHELM (HELM Audio) data3.1%
22gemini-1.5-pro-001· AHELM (HELM Audio) data3%
23Qwen2.5-Omni 7B· AHELM (HELM Audio) data3%
24Gemini 2.5 Pro Preview 05-06subject: mix· AHELM (HELM Audio) data3%
25GPT-4o mini Audio Preview· AHELM (HELM Audio) data2.9%
26Gemini 2.5 Flash Preview 05-20 (no think)subject: mix· AHELM (HELM Audio) data2.7%
27gemini-1.5-pro-002subject: mix· AHELM (HELM Audio) data2.7%
28Gemini 2.0 Flash Experimentalsubject: mix· AHELM (HELM Audio) data2.6%
29Gemini 2.0 Flashsubject: mix· AHELM (HELM Audio) data2.6%
30gemini-1.5-flash-002subject: mix· AHELM (HELM Audio) data2.6%
31Gemini 2.0 Flash Litesubject: mix· AHELM (HELM Audio) data2.5%
32GPT-4o Audio Previewsubject: mix· AHELM (HELM Audio) data2.5%
33gemini-1.5-flash-001subject: mix· AHELM (HELM Audio) data2.4%
34Qwen2.5-Omni 7Bsubject: mix· AHELM (HELM Audio) data2.4%
35gemini-1.5-pro-001subject: mix· AHELM (HELM Audio) data2.3%
36GPT-4o mini Audio Previewsubject: mix· AHELM (HELM Audio) data2.1%