Benchmark
FormosanBench MT BLEU (Mandarin→Paiwan, zero-shot)
Multilingual benchmarks compare model performance across languages. Higher scores are better.
- Models measured
- 5
- Scores recorded
- 5
- Sources
- 1
- IQ weight
- 1.04
Score distribution
- 0.001
- Mean
- 0.001
- Median
- 0.001
- Std dev
- 0
- 10th percentile
- 0.004
- 90th percentile
- 0
- Min
This benchmark has a weight of 1.04 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | GPT-4o minizero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0.0036 |
| 2 | Gemma 1.1 7B ITzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0.0019 |
| 3 | Llama 3.1 70Bzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0.0005 |
| 4 | Mistral (7B) Instruct v0.3zero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0.0002 |
| 5 | Llama 3.1 8Bzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data | 0 |
Data credits: Lin, Chen and Zhang, FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models, arXiv:2506.21563v1, CC BY 4.0. Scores extracted from Tables 2-4. (licence)