Skip to content

Benchmark

AHELM: FLEURS - WER

Scores are percentages. Lower is better.

Models measured
12
Scores recorded
84
Sources
1
IQ weight
1.87

Score distribution

0.826%
Mean
0.327%
Median
1.2%
Std dev
0.053%
10th percentile
2.1%
90th percentile
0.0359%
Min
0.0359%7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.87 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Pro Preview 05-06language: Finnish· AHELM (HELM Audio) data0.0359%
2Gemini 2.5 Pro Preview 05-06language: English· AHELM (HELM Audio) data0.0396%
3gemini-1.5-pro-002language: English· AHELM (HELM Audio) data0.0415%
4Gemini 2.0 Flashlanguage: English· AHELM (HELM Audio) data0.0493%
5Gemini 2.0 Flash Experimentallanguage: English· AHELM (HELM Audio) data0.05%
6gemini-1.5-flash-002language: English· AHELM (HELM Audio) data0.0508%
7GPT-4o mini Audio Previewlanguage: English· AHELM (HELM Audio) data0.052%
8Gemini 2.0 Flash Litelanguage: English· AHELM (HELM Audio) data0.0522%
9gemini-1.5-pro-001language: English· AHELM (HELM Audio) data0.0528%
10gemini-1.5-pro-002language: Finnish· AHELM (HELM Audio) data0.053%
11GPT-4o Audio Previewlanguage: English· AHELM (HELM Audio) data0.0565%
12Qwen2.5-Omni 7Blanguage: English· AHELM (HELM Audio) data0.0568%
13Gemini 2.0 Flash Experimentallanguage: Finnish· AHELM (HELM Audio) data0.06%
14Gemini 2.0 Flashlanguage: Finnish· AHELM (HELM Audio) data0.0615%
15Gemini 2.5 Flash Preview 05-20 (no think)language: English· AHELM (HELM Audio) data0.0633%
16Gemini 2.0 Flash Litelanguage: Finnish· AHELM (HELM Audio) data0.0814%
17gemini-1.5-pro-001language: Finnish· AHELM (HELM Audio) data0.0855%
18Gemini 2.5 Flash Preview 05-20 (no think)language: Finnish· AHELM (HELM Audio) data0.0869%
19gemini-1.5-flash-002language: Finnish· AHELM (HELM Audio) data0.115%
20gemini-1.5-flash-001language: English· AHELM (HELM Audio) data0.122%
21GPT-4o mini Audio Previewlanguage: Finnish· AHELM (HELM Audio) data0.16%
22Gemini 2.5 Pro Preview 05-06language: Hebrew· AHELM (HELM Audio) data0.162%
23Gemini 2.5 Pro Preview 05-06language: Zulu· AHELM (HELM Audio) data0.168%
24Gemini 2.5 Pro Preview 05-06language: Bengali· AHELM (HELM Audio) data0.183%
25gemini-1.5-pro-001language: Bengali· AHELM (HELM Audio) data0.19%
26Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data0.211%
27Gemini 2.0 Flash Experimentallanguage: Hebrew· AHELM (HELM Audio) data0.216%
28Gemini 2.0 Flashlanguage: Hebrew· AHELM (HELM Audio) data0.216%
29Gemini 2.5 Flash Preview 05-20 (no think)language: Bengali· AHELM (HELM Audio) data0.216%
30gemini-1.5-pro-002language: Bengali· AHELM (HELM Audio) data0.219%
31gemini-1.5-flash-001language: Bengali· AHELM (HELM Audio) data0.221%
32gemini-1.5-pro-002language: Hebrew· AHELM (HELM Audio) data0.228%
33gemini-1.5-flash-001language: Finnish· AHELM (HELM Audio) data0.238%
34Gemini 2.0 Flashlanguage: Bengali· AHELM (HELM Audio) data0.238%
35Gemini 2.0 Flash Experimentallanguage: Bengali· AHELM (HELM Audio) data0.239%
36gemini-1.5-flash-002language: Bengali· AHELM (HELM Audio) data0.265%
37Gemini 2.0 Flash Litelanguage: Hebrew· AHELM (HELM Audio) data0.272%
38Gemini 2.0 Flash Litelanguage: Bengali· AHELM (HELM Audio) data0.275%
39gemini-1.5-pro-001language: Hebrew· AHELM (HELM Audio) data0.276%
40GPT-4o Audio Previewlanguage: Finnish· AHELM (HELM Audio) data0.302%
41gemini-1.5-flash-002language: Hebrew· AHELM (HELM Audio) data0.32%
42Gemini 2.0 Flash Experimentallanguage: Zulu· AHELM (HELM Audio) data0.322%
43Gemini 2.0 Flashlanguage: Zulu· AHELM (HELM Audio) data0.327%
44gemini-1.5-flash-001language: Hebrew· AHELM (HELM Audio) data0.341%
45gemini-1.5-pro-002· AHELM (HELM Audio) data0.342%
46Gemini 2.0 Flash Litelanguage: Zulu· AHELM (HELM Audio) data0.379%
47GPT-4o mini Audio Previewlanguage: Hebrew· AHELM (HELM Audio) data0.418%
48GPT-4o mini Audio Previewlanguage: Bengali· AHELM (HELM Audio) data0.429%
49Gemini 2.0 Flash Lite· AHELM (HELM Audio) data0.443%
50gemini-1.5-flash-002· AHELM (HELM Audio) data0.461%

Showing 1–50 of 84 results.