Skip to content

Benchmark

AHELM: Air-Bench Chat (knowledge subsets) - GPT-4o Judge Critique

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages.

Models measured
12
Scores recorded
36
Sources
1
IQ weight
2.62

Score distribution

2.9%
Mean
2.9%
Median
0.316%
Std dev
2.4%
10th percentile
3.3%
90th percentile
2.2%
Min
2.2%3.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.62 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Pro Preview 05-06subject: music· AHELM (HELM Audio) data3.5%
2Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data3.4%
3Gemini 2.5 Pro Preview 05-06subject: sound· AHELM (HELM Audio) data3.3%
4Gemini 2.5 Flash Preview 05-20 (no think)subject: music· AHELM (HELM Audio) data3.3%
5Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data3.2%
6Gemini 2.5 Flash Preview 05-20 (no think)subject: sound· AHELM (HELM Audio) data3.1%
7GPT-4o Audio Previewsubject: sound· AHELM (HELM Audio) data3.1%
8Gemini 2.0 Flashsubject: music· AHELM (HELM Audio) data3.1%
9Gemini 2.0 Flash Experimentalsubject: music· AHELM (HELM Audio) data3.1%
10Gemini 2.0 Flash· AHELM (HELM Audio) data3%
11GPT-4o Audio Preview· AHELM (HELM Audio) data3%
12Gemini 2.0 Flash Litesubject: music· AHELM (HELM Audio) data3%
13Gemini 2.0 Flashsubject: sound· AHELM (HELM Audio) data3%
14Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data3%
15Gemini 2.0 Flash Experimentalsubject: sound· AHELM (HELM Audio) data3%
16gemini-1.5-pro-002subject: music· AHELM (HELM Audio) data3%
17GPT-4o Audio Previewsubject: music· AHELM (HELM Audio) data3%
18Gemini 2.0 Flash Lite· AHELM (HELM Audio) data2.9%
19gemini-1.5-flash-002subject: music· AHELM (HELM Audio) data2.9%
20GPT-4o mini Audio Previewsubject: sound· AHELM (HELM Audio) data2.9%
21gemini-1.5-pro-002· AHELM (HELM Audio) data2.9%
22Qwen2.5-Omni 7Bsubject: sound· AHELM (HELM Audio) data2.8%
23gemini-1.5-flash-002· AHELM (HELM Audio) data2.8%
24Gemini 2.0 Flash Litesubject: sound· AHELM (HELM Audio) data2.8%
25GPT-4o mini Audio Preview· AHELM (HELM Audio) data2.8%
26gemini-1.5-pro-002subject: sound· AHELM (HELM Audio) data2.7%
27gemini-1.5-flash-002subject: sound· AHELM (HELM Audio) data2.7%
28GPT-4o mini Audio Previewsubject: music· AHELM (HELM Audio) data2.7%
29Qwen2.5-Omni 7B· AHELM (HELM Audio) data2.7%
30gemini-1.5-flash-001subject: music· AHELM (HELM Audio) data2.5%
31Qwen2.5-Omni 7Bsubject: music· AHELM (HELM Audio) data2.5%
32gemini-1.5-flash-001· AHELM (HELM Audio) data2.4%
33gemini-1.5-pro-001subject: sound· AHELM (HELM Audio) data2.3%
34gemini-1.5-flash-001subject: sound· AHELM (HELM Audio) data2.3%
35gemini-1.5-pro-001· AHELM (HELM Audio) data2.3%
36gemini-1.5-pro-001subject: music· AHELM (HELM Audio) data2.2%