Skip to content

Benchmark

AHELM: LibriSpeech Fairness - WER

Scores are percentages. Lower is better.

Models measured
12
Scores recorded
36
Sources
1
IQ weight
1.58

Score distribution

8.6%
Mean
6%
Median
6.8%
Std dev
3.8%
10th percentile
15.3%
90th percentile
3.3%
Min
3.3%29.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.58 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Pro Preview 05-06gender: male· AHELM (HELM Audio) data3.3%
2Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data3.6%
3Gemini 2.0 Flash Experimentalgender: male· AHELM (HELM Audio) data3.7%
4Gemini 2.0 Flashgender: male· AHELM (HELM Audio) data3.7%
5Gemini 2.5 Pro Preview 05-06gender: female· AHELM (HELM Audio) data3.8%
6Gemini 2.0 Flash Litegender: male· AHELM (HELM Audio) data4.1%
7Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data4.1%
8Gemini 2.0 Flash· AHELM (HELM Audio) data4.1%
9Gemini 2.0 Flash Experimentalgender: female· AHELM (HELM Audio) data4.5%
10Gemini 2.0 Flashgender: female· AHELM (HELM Audio) data4.5%
11Gemini 2.0 Flash Lite· AHELM (HELM Audio) data4.6%
12gemini-1.5-pro-002gender: male· AHELM (HELM Audio) data4.8%
13gemini-1.5-pro-002· AHELM (HELM Audio) data4.9%
14gemini-1.5-pro-002gender: female· AHELM (HELM Audio) data4.9%
15gemini-1.5-flash-002gender: female· AHELM (HELM Audio) data5%
16Gemini 2.0 Flash Litegender: female· AHELM (HELM Audio) data5.1%
17gemini-1.5-flash-002· AHELM (HELM Audio) data5.5%
18gemini-1.5-flash-002gender: male· AHELM (HELM Audio) data5.9%
19Gemini 2.5 Flash Preview 05-20 (no think)gender: male· AHELM (HELM Audio) data6%
20gemini-1.5-pro-001gender: female· AHELM (HELM Audio) data7%
21Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data7.1%
22gemini-1.5-pro-001· AHELM (HELM Audio) data7.2%
23gemini-1.5-pro-001gender: male· AHELM (HELM Audio) data7.5%
24Qwen2.5-Omni 7Bgender: male· AHELM (HELM Audio) data7.5%
25Gemini 2.5 Flash Preview 05-20 (no think)gender: female· AHELM (HELM Audio) data8.2%
26Qwen2.5-Omni 7B· AHELM (HELM Audio) data8.5%
27GPT-4o Audio Previewgender: male· AHELM (HELM Audio) data9.2%
28Qwen2.5-Omni 7Bgender: female· AHELM (HELM Audio) data9.5%
29GPT-4o Audio Preview· AHELM (HELM Audio) data10.6%
30GPT-4o Audio Previewgender: female· AHELM (HELM Audio) data12%
31GPT-4o mini Audio Previewgender: male· AHELM (HELM Audio) data15.3%
32GPT-4o mini Audio Preview· AHELM (HELM Audio) data15.3%
33GPT-4o mini Audio Previewgender: female· AHELM (HELM Audio) data15.3%
34gemini-1.5-flash-001gender: male· AHELM (HELM Audio) data26.4%
35gemini-1.5-flash-001· AHELM (HELM Audio) data28.2%
36gemini-1.5-flash-001gender: female· AHELM (HELM Audio) data29.9%