Skip to content

Benchmark

AHELM: CoVost-2 - BLEU

Scores are percentages.

Models measured
12
Scores recorded
36
Sources
1
IQ weight
0.20

Score distribution

31.3%
Mean
31.8%
Median
8.5%
Std dev
20.5%
10th percentile
42.6%
90th percentile
18%
Min
18%43.8%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.20 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1gemini-1.5-pro-002source_language: Spanish, target_language: English· AHELM (HELM Audio) data43.8%
2Gemini 2.5 Pro Preview 05-06source_language: Spanish, target_language: English· AHELM (HELM Audio) data43.8%
3gemini-1.5-pro-001source_language: Spanish, target_language: English· AHELM (HELM Audio) data43.4%
4Gemini 2.0 Flashsource_language: Spanish, target_language: English· AHELM (HELM Audio) data42.6%
5gemini-1.5-flash-002source_language: Spanish, target_language: English· AHELM (HELM Audio) data42.3%
6Gemini 2.5 Flash Preview 05-20 (no think)source_language: Spanish, target_language: English· AHELM (HELM Audio) data42%
7gemini-1.5-flash-001source_language: Spanish, target_language: English· AHELM (HELM Audio) data41.9%
8GPT-4o Audio Previewsource_language: Spanish, target_language: English· AHELM (HELM Audio) data41.6%
9Gemini 2.0 Flash Experimentalsource_language: Spanish, target_language: English· AHELM (HELM Audio) data41.4%
10Gemini 2.0 Flash Litesource_language: Spanish, target_language: English· AHELM (HELM Audio) data41.2%
11GPT-4o mini Audio Previewsource_language: Spanish, target_language: English· AHELM (HELM Audio) data38.7%
12Gemini 2.5 Pro Preview 05-06· AHELM (HELM Audio) data35.7%
13Gemini 2.0 Flash· AHELM (HELM Audio) data33.5%
14Gemini 2.5 Flash Preview 05-20 (no think)· AHELM (HELM Audio) data33.4%
15gemini-1.5-pro-002· AHELM (HELM Audio) data33%
16Gemini 2.0 Flash Experimental· AHELM (HELM Audio) data32.9%
17gemini-1.5-pro-001· AHELM (HELM Audio) data32.7%
18Gemini 2.0 Flash Lite· AHELM (HELM Audio) data31.8%
19GPT-4o Audio Preview· AHELM (HELM Audio) data31.6%
20gemini-1.5-flash-001· AHELM (HELM Audio) data30.7%
21gemini-1.5-flash-002· AHELM (HELM Audio) data30.6%
22GPT-4o mini Audio Preview· AHELM (HELM Audio) data29.3%
23Gemini 2.5 Pro Preview 05-06source_language: Chinese, target_language: English· AHELM (HELM Audio) data27.6%
24Gemini 2.5 Flash Preview 05-20 (no think)source_language: Chinese, target_language: English· AHELM (HELM Audio) data24.7%
25Gemini 2.0 Flash Experimentalsource_language: Chinese, target_language: English· AHELM (HELM Audio) data24.4%
26Gemini 2.0 Flashsource_language: Chinese, target_language: English· AHELM (HELM Audio) data24.3%
27Qwen2.5-Omni 7Bsource_language: Spanish, target_language: English· AHELM (HELM Audio) data23%
28Gemini 2.0 Flash Litesource_language: Chinese, target_language: English· AHELM (HELM Audio) data22.4%
29gemini-1.5-pro-002source_language: Chinese, target_language: English· AHELM (HELM Audio) data22.2%
30gemini-1.5-pro-001source_language: Chinese, target_language: English· AHELM (HELM Audio) data22%
31GPT-4o Audio Previewsource_language: Chinese, target_language: English· AHELM (HELM Audio) data21.5%
32Qwen2.5-Omni 7B· AHELM (HELM Audio) data20.5%
33GPT-4o mini Audio Previewsource_language: Chinese, target_language: English· AHELM (HELM Audio) data19.8%
34gemini-1.5-flash-001source_language: Chinese, target_language: English· AHELM (HELM Audio) data19.5%
35gemini-1.5-flash-002source_language: Chinese, target_language: English· AHELM (HELM Audio) data18.9%
36Qwen2.5-Omni 7Bsource_language: Chinese, target_language: English· AHELM (HELM Audio) data18%