Skip to content

Benchmark

Pashto ASR CV24_filtered CER (zero-shot)

Scores are percentages. Lower is better.

Models measured
7
Scores recorded
7
Sources
1
IQ weight
1.03

Score distribution

171.5%
Mean
81.6%
Median
142.3%
Std dev
73.7%
10th percentile
351.9%
90th percentile
44.6%
Min
44.6%428.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.03 for speech models, measured over 7 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Whisper Large V3table: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data44.6%
2Whisper Smalltable: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data73.7%
3Whisper Large v2table: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data78.6%
4Whisper Large V3 Turbotable: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data81.6%
5Whisper Basetable: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data141.6%
6Whisper Tinytable: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data351.9%
7Whisper Mediumtable: II, dataset: CV24_filtered, metric: CER, protocol: zero-shot· Pashto ASR paper data428.2%

Data credits: Hanif Rahman. Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation, arXiv:2604.04598v1. Numeric/protocol extract from Tables II, III, V and VI; CC BY 4.0 article grant verified. (licence)