Skip to content

Benchmark

Pashto ASR FLEURS ps_af WER (zero-shot)

Scores are percentages. Lower is better.

Models measured
7
Scores recorded
7
Sources
1
IQ weight
1.11

Score distribution

142.5%
Mean
100.6%
Median
73.3%
Std dev
91.8%
10th percentile
204.6%
90th percentile
89.8%
Min
89.8%297%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.11 for speech models, measured over 7 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Whisper Large V3table: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data89.8%
2Whisper Large V3 Turbotable: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data91.8%
3Whisper Large v2table: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data95.7%
4Whisper Smalltable: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data100.6%
5Whisper Basetable: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data118.2%
6Whisper Mediumtable: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data204.6%
7Whisper Tinytable: II, dataset: FLEURS ps_af, metric: WER, protocol: zero-shot· Pashto ASR paper data297%

Data credits: Hanif Rahman. Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation, arXiv:2604.04598v1. Numeric/protocol extract from Tables II, III, V and VI; CC BY 4.0 article grant verified. (licence)