Skip to content

Benchmark

AHELM: Multilingual Librispeech - WER

Scores are percentages. Lower is better.

Models measured
12
Scores recorded
96
Sources
1
IQ weight
1.90

Score distribution

0.165%
Mean
0.073%
Median
0.325%
Std dev
0.046%
10th percentile
0.269%
90th percentile
0.03%
Min
0.03%2.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.90 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51gemini-1.5-flash-002language: Dutch· AHELM (HELM Audio) data0.0748%
52gemini-1.5-pro-002language: Italian· AHELM (HELM Audio) data0.0748%
53gemini-1.5-flash-001language: French· AHELM (HELM Audio) data0.0775%
54Gemini 2.5 Flash Preview 05-20 (no think)language: Dutch· AHELM (HELM Audio) data0.0778%
55GPT-4o mini Audio Previewlanguage: Spanish· AHELM (HELM Audio) data0.0793%
56gemini-1.5-pro-001language: German· AHELM (HELM Audio) data0.0801%
57gemini-1.5-flash-001language: Portuguese· AHELM (HELM Audio) data0.0842%
58gemini-1.5-flash-001language: Dutch· AHELM (HELM Audio) data0.0851%
59gemini-1.5-flash-001language: German· AHELM (HELM Audio) data0.0864%
60gemini-1.5-flash-001· AHELM (HELM Audio) data0.0881%
61gemini-1.5-pro-001language: Dutch· AHELM (HELM Audio) data0.0914%
62Gemini 2.0 Flashlanguage: Italian· AHELM (HELM Audio) data0.0955%
63Gemini 2.0 Flash Experimentallanguage: Italian· AHELM (HELM Audio) data0.096%
64GPT-4o mini Audio Previewlanguage: French· AHELM (HELM Audio) data0.0967%
65gemini-1.5-pro-001language: Italian· AHELM (HELM Audio) data0.0986%
66GPT-4o mini Audio Previewlanguage: German· AHELM (HELM Audio) data0.102%
67Gemini 2.0 Flash Litelanguage: Italian· AHELM (HELM Audio) data0.103%
68gemini-1.5-flash-002language: Italian· AHELM (HELM Audio) data0.112%
69GPT-4o mini Audio Previewlanguage: Portuguese· AHELM (HELM Audio) data0.116%
70Gemini 2.5 Flash Preview 05-20 (no think)language: Italian· AHELM (HELM Audio) data0.123%
71GPT-4o mini Audio Preview· AHELM (HELM Audio) data0.123%
72GPT-4o Audio Previewlanguage: Spanish· AHELM (HELM Audio) data0.126%
73GPT-4o Audio Previewlanguage: Italian· AHELM (HELM Audio) data0.132%
74GPT-4o mini Audio Previewlanguage: Dutch· AHELM (HELM Audio) data0.133%
75gemini-1.5-flash-001language: Italian· AHELM (HELM Audio) data0.142%
76GPT-4o mini Audio Previewlanguage: Polish· AHELM (HELM Audio) data0.142%
77GPT-4o Audio Previewlanguage: Portuguese· AHELM (HELM Audio) data0.149%
78GPT-4o Audio Preview· AHELM (HELM Audio) data0.162%
79GPT-4o Audio Previewlanguage: German· AHELM (HELM Audio) data0.164%
80GPT-4o Audio Previewlanguage: French· AHELM (HELM Audio) data0.164%
81GPT-4o Audio Previewlanguage: Polish· AHELM (HELM Audio) data0.172%
82GPT-4o mini Audio Previewlanguage: Italian· AHELM (HELM Audio) data0.191%
83Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data0.198%
84Qwen2.5-Omni 7Blanguage: Spanish· AHELM (HELM Audio) data0.205%
85GPT-4o Audio Previewlanguage: Dutch· AHELM (HELM Audio) data0.228%
86Qwen2.5-Omni 7Blanguage: Italian· AHELM (HELM Audio) data0.24%
87Qwen2.5-Omni 7Blanguage: Portuguese· AHELM (HELM Audio) data0.269%
88Qwen2.5-Omni 7Blanguage: French· AHELM (HELM Audio) data0.293%
89Qwen2.5-Omni 7Blanguage: German· AHELM (HELM Audio) data0.343%
90Qwen2.5-Omni 7B· AHELM (HELM Audio) data0.416%
91Qwen2.5-Omni 7Blanguage: Dutch· AHELM (HELM Audio) data0.535%
92Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data0.603%
93Qwen2.5-Omni 7Blanguage: Polish· AHELM (HELM Audio) data1%
94Gemini 2.5 Pro Preview 05-06language: Italian· AHELM (HELM Audio) data1.1%
95Gemini 2.5 Flash Preview 05-20 (no think)language: Spanish· AHELM (HELM Audio) data1.1%
96Gemini 2.5 Flash Preview 05-20 (no think)language: German· AHELM (HELM Audio) data2.7%

Showing 51–96 of 96 results.