Skip to content

Benchmark

AHELM: FLEURS Fairness - WER

Scores are percentages. Lower is better.

Models measured
12
Scores recorded
36
Sources
1
IQ weight
1.86

Score distribution

5.8%
Mean
5.2%
Median
2.1%
Std dev
4.2%
10th percentile
6.8%
90th percentile
3.8%
Min
3.8%12.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.86 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Pro Preview 05-06gender: female· AHELM (HELM Audio) data3.8%
2Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data4%
3gemini-1.5-pro-002gender: female· AHELM (HELM Audio) data4.1%
4Gemini 2.5 Pro Preview 05-06gender: male· AHELM (HELM Audio) data4.2%
5gemini-1.5-pro-002· AHELM (HELM Audio) data4.2%
6gemini-1.5-pro-002gender: male· AHELM (HELM Audio) data4.4%
7Gemini 2.0 Flashgender: female· AHELM (HELM Audio) data4.7%
8Gemini 2.0 Flash Experimentalgender: female· AHELM (HELM Audio) data4.7%
9gemini-1.5-flash-002gender: male· AHELM (HELM Audio) data4.9%
10Gemini 2.0 Flash· AHELM (HELM Audio) data5%
11Qwen2.5-Omni 7Bgender: female· AHELM (HELM Audio) data5.1%
12gemini-1.5-flash-002· AHELM (HELM Audio) data5.1%
13Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data5.1%
14gemini-1.5-pro-001gender: female· AHELM (HELM Audio) data5.1%
15Gemini 2.0 Flash Litegender: female· AHELM (HELM Audio) data5.1%
16GPT-4o mini Audio Previewgender: male· AHELM (HELM Audio) data5.2%
17GPT-4o mini Audio Preview· AHELM (HELM Audio) data5.2%
18gemini-1.5-flash-002gender: female· AHELM (HELM Audio) data5.2%
19GPT-4o mini Audio Previewgender: female· AHELM (HELM Audio) data5.2%
20Gemini 2.0 Flash Lite· AHELM (HELM Audio) data5.3%
21Gemini 2.0 Flash Litegender: male· AHELM (HELM Audio) data5.4%
22gemini-1.5-pro-001· AHELM (HELM Audio) data5.4%
23Gemini 2.0 Flash Experimentalgender: male· AHELM (HELM Audio) data5.4%
24Gemini 2.0 Flashgender: male· AHELM (HELM Audio) data5.4%
25GPT-4o Audio Previewgender: female· AHELM (HELM Audio) data5.4%
26Gemini 2.5 Flash Preview 05-20 (no think)gender: male· AHELM (HELM Audio) data5.7%
27gemini-1.5-pro-001gender: male· AHELM (HELM Audio) data5.7%
28GPT-4o Audio Preview· AHELM (HELM Audio) data5.8%
29Qwen2.5-Omni 7B· AHELM (HELM Audio) data5.9%
30GPT-4o Audio Previewgender: male· AHELM (HELM Audio) data6.1%
31Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data6.2%
32Gemini 2.5 Flash Preview 05-20 (no think)gender: female· AHELM (HELM Audio) data6.7%
33Qwen2.5-Omni 7Bgender: male· AHELM (HELM Audio) data6.8%
34gemini-1.5-flash-001gender: female· AHELM (HELM Audio) data11.9%
35gemini-1.5-flash-001· AHELM (HELM Audio) data12.3%
36gemini-1.5-flash-001gender: male· AHELM (HELM Audio) data12.8%