Skip to content

Benchmark

AHELM: Multilingual Librispeech - WER

Scores are percentages. Lower is better.

Models measured
12
Scores recorded
96
Sources
1
IQ weight
1.90

Score distribution

0.165%
Mean
0.073%
Median
0.325%
Std dev
0.046%
10th percentile
0.269%
90th percentile
0.03%
Min
0.03%2.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.90 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Pro Preview 05-06language: Polish· AHELM (HELM Audio) data0.03%
2Gemini 2.5 Pro Preview 05-06language: Spanish· AHELM (HELM Audio) data0.0328%
3Gemini 2.0 Flash Experimentallanguage: Spanish· AHELM (HELM Audio) data0.0387%
4Gemini 2.0 Flashlanguage: Spanish· AHELM (HELM Audio) data0.0388%
5gemini-1.5-pro-002language: Spanish· AHELM (HELM Audio) data0.04%
6gemini-1.5-pro-002language: Polish· AHELM (HELM Audio) data0.0406%
7Gemini 2.5 Pro Preview 05-06language: Portuguese· AHELM (HELM Audio) data0.0413%
8Gemini 2.0 Flashlanguage: Polish· AHELM (HELM Audio) data0.0421%
9Gemini 2.0 Flash Experimentallanguage: Polish· AHELM (HELM Audio) data0.0423%
10Gemini 2.0 Flash Litelanguage: Spanish· AHELM (HELM Audio) data0.0426%
11gemini-1.5-flash-002language: Spanish· AHELM (HELM Audio) data0.0463%
12Gemini 2.5 Pro Preview 05-06language: German· AHELM (HELM Audio) data0.0479%
13gemini-1.5-pro-002language: Portuguese· AHELM (HELM Audio) data0.0495%
14Gemini 2.0 Flashlanguage: Portuguese· AHELM (HELM Audio) data0.0522%
15Gemini 2.0 Flash Experimentallanguage: Portuguese· AHELM (HELM Audio) data0.0523%
16gemini-1.5-pro-002language: French· AHELM (HELM Audio) data0.053%
17Gemini 2.0 Flash Litelanguage: Polish· AHELM (HELM Audio) data0.0533%
18gemini-1.5-pro-002· AHELM (HELM Audio) data0.0539%
19gemini-1.5-pro-002language: German· AHELM (HELM Audio) data0.0553%
20gemini-1.5-pro-001language: Polish· AHELM (HELM Audio) data0.0554%
21gemini-1.5-pro-001language: Spanish· AHELM (HELM Audio) data0.0558%
22Gemini 2.5 Flash Preview 05-20 (no think)language: Polish· AHELM (HELM Audio) data0.0571%
23Gemini 2.5 Pro Preview 05-06language: Dutch· AHELM (HELM Audio) data0.0583%
24Gemini 2.0 Flash Experimentallanguage: German· AHELM (HELM Audio) data0.0599%
25Gemini 2.0 Flashlanguage: German· AHELM (HELM Audio) data0.0599%
26Gemini 2.0 Flash· AHELM (HELM Audio) data0.06%
27Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data0.06%
28gemini-1.5-flash-002language: Polish· AHELM (HELM Audio) data0.0604%
29gemini-1.5-pro-001language: Portuguese· AHELM (HELM Audio) data0.0618%
30Gemini 2.5 Pro Preview 05-06language: French· AHELM (HELM Audio) data0.0635%
31Gemini 2.0 Flash Litelanguage: Portuguese· AHELM (HELM Audio) data0.0636%
32gemini-1.5-pro-002language: Dutch· AHELM (HELM Audio) data0.0641%
33Gemini 2.0 Flash Experimentallanguage: French· AHELM (HELM Audio) data0.0653%
34gemini-1.5-flash-002language: German· AHELM (HELM Audio) data0.0654%
35gemini-1.5-pro-001language: French· AHELM (HELM Audio) data0.0655%
36Gemini 2.0 Flashlanguage: French· AHELM (HELM Audio) data0.0655%
37Gemini 2.0 Flash Experimentallanguage: Dutch· AHELM (HELM Audio) data0.0657%
38Gemini 2.0 Flashlanguage: Dutch· AHELM (HELM Audio) data0.0658%
39Gemini 2.0 Flash Litelanguage: German· AHELM (HELM Audio) data0.0662%
40gemini-1.5-flash-002language: French· AHELM (HELM Audio) data0.0668%
41Gemini 2.0 Flash Lite· AHELM (HELM Audio) data0.0673%
42Gemini 2.0 Flash Litelanguage: French· AHELM (HELM Audio) data0.0683%
43gemini-1.5-flash-001language: Spanish· AHELM (HELM Audio) data0.0694%
44Gemini 2.5 Flash Preview 05-20 (no think)language: French· AHELM (HELM Audio) data0.0694%
45gemini-1.5-flash-002language: Portuguese· AHELM (HELM Audio) data0.0695%
46gemini-1.5-flash-002· AHELM (HELM Audio) data0.0707%
47gemini-1.5-flash-001language: Polish· AHELM (HELM Audio) data0.072%
48gemini-1.5-pro-001· AHELM (HELM Audio) data0.0727%
49Gemini 2.0 Flash Litelanguage: Dutch· AHELM (HELM Audio) data0.0734%
50Gemini 2.5 Flash Preview 05-20 (no think)language: Portuguese· AHELM (HELM Audio) data0.0735%

Showing 1–50 of 96 results.