Skip to content

Benchmark

Pashto ASR CV24_filtered WER (zero-shot)

Scores are percentages. Lower is better.

Models measured
7
Scores recorded
7
Sources
1
IQ weight
1.09

Score distribution

191.5%
Mean
120.4%
Median
124.2%
Std dev
107.2%
10th percentile
274.6%
90th percentile
95.4%
Min
95.4%461.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.09 for speech models, measured over 7 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Whisper Large V3table: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data95.4%
2Whisper Large v2table: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data107.2%
3Whisper Large V3 Turbotable: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data110.2%
4Whisper Smalltable: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data120.4%
5Whisper Basetable: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data171.4%
6Whisper Tinytable: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data274.6%
7Whisper Mediumtable: II, dataset: CV24_filtered, metric: WER, protocol: zero-shot· Pashto ASR paper data461.2%

Data credits: Hanif Rahman. Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation, arXiv:2604.04598v1. Numeric/protocol extract from Tables II, III, V and VI; CC BY 4.0 article grant verified. (licence)