Skip to content

Benchmark

AHELM: Air-Bench Foundation - EM

Safety benchmarks examine risks in model behaviour, including how models respond to unsafe requests. Scores are percentages.

Models measured
12
Scores recorded
56
Sources
1
IQ weight
1.10

Score distribution

59.3%
Mean
57.5%
Median
11.9%
Std dev
44.3%
10th percentile
77%
90th percentile
32.7%
Min
32.7%80.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.10 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Qwen2.5-Omni 7Bsubject: music_genre_recognition· AHELM (HELM Audio) data80.8%
2Qwen2.5-Omni 7Bsubject: music_qa· AHELM (HELM Audio) data78.7%
3Gemini 2.5 Pro Preview 05-06subject: music_genre_recognition· AHELM (HELM Audio) data77.7%
4Gemini 2.0 Flashsubject: music_genera_recognition· AHELM (HELM Audio) data77%
5Gemini 2.0 Flashsubject: music_genre_recognition· AHELM (HELM Audio) data77%
6Gemini 2.0 Flash Experimentalsubject: music_genera_recognition· AHELM (HELM Audio) data77%
7Gemini 2.0 Flash Experimentalsubject: music_genre_recognition· AHELM (HELM Audio) data77%
8Gemini 2.0 Flash Litesubject: music_genre_recognition· AHELM (HELM Audio) data76.3%
9Qwen2.5-Omni 7B· AHELM (HELM Audio) data74.3%
10gemini-1.5-pro-002subject: music_qa· AHELM (HELM Audio) data70.6%
11Gemini 2.5 Pro Preview 05-06subject: music_qa· AHELM (HELM Audio) data70.3%
12Gemini 2.0 Flash Experimentalsubject: music_qa· AHELM (HELM Audio) data69.8%
13Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data69.8%
14Gemini 2.0 Flash· AHELM (HELM Audio) data69.7%
15Gemini 2.0 Flashsubject: music_qa· AHELM (HELM Audio) data69.7%
16Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data68.3%
17Gemini 2.5 Flash Preview 05-20 (no think)subject: music_genre_recognition· AHELM (HELM Audio) data66.6%
18Gemini 2.0 Flash Litesubject: music_qa· AHELM (HELM Audio) data64.4%
19Gemini 2.0 Flash Lite· AHELM (HELM Audio) data64.1%
20gemini-1.5-flash-002subject: music_qa· AHELM (HELM Audio) data64%
21Qwen2.5-Omni 7Bsubject: music_instrument_classification· AHELM (HELM Audio) data63.5%
22Gemini 2.5 Flash Preview 05-20 (no think)subject: music_qa· AHELM (HELM Audio) data62.9%
23GPT-4o mini Audio Previewsubject: music_qa· AHELM (HELM Audio) data59.1%
24GPT-4o Audio Previewsubject: music_qa· AHELM (HELM Audio) data58.8%
25GPT-4o mini Audio Previewsubject: music_genera_recognition· AHELM (HELM Audio) data58.3%
26GPT-4o mini Audio Previewsubject: music_genre_recognition· AHELM (HELM Audio) data58.3%
27Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data57.9%
28GPT-4o Audio Previewsubject: music_genera_recognition· AHELM (HELM Audio) data57.5%
29GPT-4o Audio Previewsubject: music_genre_recognition· AHELM (HELM Audio) data57.5%
30gemini-1.5-pro-002subject: music_genera_recognition· AHELM (HELM Audio) data57.1%
31gemini-1.5-pro-002subject: music_genre_recognition· AHELM (HELM Audio) data57.1%
32Gemini 2.5 Pro Preview 05-06subject: music_instrument_classification· AHELM (HELM Audio) data57%
33gemini-1.5-pro-001subject: music_genera_recognition· AHELM (HELM Audio) data56.1%
34gemini-1.5-pro-001subject: music_genre_recognition· AHELM (HELM Audio) data56.1%
35gemini-1.5-pro-002· AHELM (HELM Audio) data55.4%
36Gemini 2.0 Flash Experimentalsubject: music_instrument_classification· AHELM (HELM Audio) data55.3%
37Gemini 2.0 Flashsubject: music_instrument_classification· AHELM (HELM Audio) data55.2%
38gemini-1.5-pro-001subject: music_qa· AHELM (HELM Audio) data54.4%
39GPT-4o mini Audio Preview· AHELM (HELM Audio) data54.1%
40gemini-1.5-flash-001subject: music_qa· AHELM (HELM Audio) data54.1%
41gemini-1.5-flash-001subject: music_genera_recognition· AHELM (HELM Audio) data53.3%
42gemini-1.5-flash-001subject: music_genre_recognition· AHELM (HELM Audio) data53.3%
43gemini-1.5-flash-002subject: music_genera_recognition· AHELM (HELM Audio) data52.7%
44gemini-1.5-flash-002subject: music_genre_recognition· AHELM (HELM Audio) data52.7%
45GPT-4o Audio Preview· AHELM (HELM Audio) data52.7%
46Gemini 2.0 Flash Litesubject: music_instrument_classification· AHELM (HELM Audio) data51.5%
47gemini-1.5-pro-001· AHELM (HELM Audio) data51.1%
48gemini-1.5-flash-002· AHELM (HELM Audio) data50.8%
49gemini-1.5-flash-001· AHELM (HELM Audio) data48.3%
50Gemini 2.5 Flash Preview 05-20 (no think)subject: music_instrument_classification· AHELM (HELM Audio) data44.3%

Showing 1–50 of 56 results.