Skip to content

Benchmark

FormosanBench summarization ROUGE-2 (Atayal, 10-shot)

Multilingual benchmarks compare model performance across languages. Scores are percentages. Higher is better.

Models measured
3
Scores recorded
3
Sources
1
IQ weight
No weight published

Score distribution

13.1%
Mean
0.52%
Median
17.8%
Std dev
0.52%
10th percentile
38.4%
90th percentile
0.52%
Min
0.52%38.4%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#ModelScore
1GPT 4o10-shot, 10 in-context examples, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data38.4%
2Llama 3.1 8B10-shot, 10 in-context examples, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.52%
3mistral-7b-v0.310-shot, 10 in-context examples, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.52%

Data credits: Lin, Chen and Zhang, FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models, arXiv:2506.21563v1, CC BY 4.0. Scores extracted from Tables 2-4. (licence)