Skip to content

Benchmark

AHELM: FLEURS - WER

Scores are percentages. Lower is better.

Models measured
12
Scores recorded
84
Sources
1
IQ weight
1.87

Score distribution

0.826%
Mean
0.327%
Median
1.2%
Std dev
0.053%
10th percentile
2.1%
90th percentile
0.0359%
Min
0.0359%7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.87 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51gemini-1.5-pro-001· AHELM (HELM Audio) data0.463%
52GPT-4o Audio Previewlanguage: Hebrew· AHELM (HELM Audio) data0.522%
53gemini-1.5-pro-002language: Zulu· AHELM (HELM Audio) data0.533%
54GPT-4o mini Audio Preview· AHELM (HELM Audio) data0.545%
55Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data0.646%
56Gemini 2.0 Flash· AHELM (HELM Audio) data0.648%
57gemini-1.5-pro-001language: Zulu· AHELM (HELM Audio) data0.677%
58Gemini 2.5 Pro Preview 05-06language: Thai· AHELM (HELM Audio) data0.677%
59GPT-4o Audio Previewlanguage: Bengali· AHELM (HELM Audio) data0.698%
60gemini-1.5-flash-001· AHELM (HELM Audio) data0.723%
61GPT-4o Audio Preview· AHELM (HELM Audio) data0.771%
62gemini-1.5-flash-002language: Zulu· AHELM (HELM Audio) data0.948%
63gemini-1.5-pro-002language: Thai· AHELM (HELM Audio) data0.977%
64GPT-4o Audio Previewlanguage: Zulu· AHELM (HELM Audio) data0.983%
65GPT-4o mini Audio Previewlanguage: Thai· AHELM (HELM Audio) data1%
66gemini-1.5-flash-002language: Thai· AHELM (HELM Audio) data1.1%
67GPT-4o mini Audio Previewlanguage: Zulu· AHELM (HELM Audio) data1.2%
68gemini-1.5-flash-001language: Zulu· AHELM (HELM Audio) data1.3%
69Qwen2.5-Omni 7Blanguage: Bengali· AHELM (HELM Audio) data1.4%
70gemini-1.5-pro-001language: Thai· AHELM (HELM Audio) data1.5%
71Qwen2.5-Omni 7Blanguage: Hebrew· AHELM (HELM Audio) data1.6%
72Gemini 2.0 Flash Litelanguage: Thai· AHELM (HELM Audio) data1.6%
73Qwen2.5-Omni 7Blanguage: Finnish· AHELM (HELM Audio) data1.6%
74Qwen2.5-Omni 7Blanguage: Zulu· AHELM (HELM Audio) data1.8%
75Qwen2.5-Omni 7B· AHELM (HELM Audio) data1.9%
76GPT-4o Audio Previewlanguage: Thai· AHELM (HELM Audio) data2.1%
77gemini-1.5-flash-001language: Thai· AHELM (HELM Audio) data2.1%
78Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data2.7%
79Gemini 2.0 Flash Experimentallanguage: Thai· AHELM (HELM Audio) data3%
80Gemini 2.0 Flashlanguage: Thai· AHELM (HELM Audio) data3%
81Gemini 2.5 Flash Preview 05-20 (no think)language: Hebrew· AHELM (HELM Audio) data3.2%
82Qwen2.5-Omni 7Blanguage: Thai· AHELM (HELM Audio) data5.2%
83Gemini 2.5 Flash Preview 05-20 (no think)language: Thai· AHELM (HELM Audio) data5.9%
84Gemini 2.5 Flash Preview 05-20 (no think)language: Zulu· AHELM (HELM Audio) data7%

Showing 51–84 of 84 results.