Benchmark
FormosanBench summarization ROUGE-2 (Atayal, 10-shot)
Multilingual benchmarks compare model performance across languages. Scores are percentages. Higher is better.
- Models measured
- 3
- Scores recorded
- 3
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 13.1%
- Mean
- 0.52%
- Median
- 17.8%
- Std dev
- 0.52%
- 10th percentile
- 38.4%
- 90th percentile
- 0.52%
- Min
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | GPT 4o10-shot, 10 in-context examples, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 38.4% |
| 2 | Llama 3.1 8B10-shot, 10 in-context examples, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0.52% |
| 3 | mistral-7b-v0.310-shot, 10 in-context examples, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0.52% |
Data credits: Lin, Chen and Zhang, FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models, arXiv:2506.21563v1, CC BY 4.0. Scores extracted from Tables 2-4. (licence)