Skip to content

Benchmark

FormosanBench summarization ROUGE-2 (Paiwan, zero-shot)

Multilingual benchmarks compare model performance across languages. Scores are percentages. Higher is better.

Models measured
2
Scores recorded
2
Sources
1
IQ weight
No weight published

Score distribution

9.7%
Mean
13.2%
Median
3.5%
Std dev
6.3%
10th percentile
13.2%
90th percentile
6.3%
Min
6.3%13.2%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#ModelScore
1mistral-7b-v0.3zero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data13.2%
2GPT 4ozero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data6.3%

Data credits: Lin, Chen and Zhang, FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models, arXiv:2506.21563v1, CC BY 4.0. Scores extracted from Tables 2-4. (licence)