Skip to content

Benchmark

FormosanBench MT BLEU (Mandarin→Paiwan, zero-shot)

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models measured
5
Scores recorded
5
Sources
1
IQ weight
1.04

Score distribution

0.001
Mean
0.001
Median
0.001
Std dev
0
10th percentile
0.004
90th percentile
0
Min

This benchmark has a weight of 1.04 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT-4o minizero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0036
2Gemma 1.1 7B ITzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0019
3Llama 3.1 70Bzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0005
4Mistral (7B) Instruct v0.3zero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0002
5Llama 3.1 8Bzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0

Data credits: Lin, Chen and Zhang, FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models, arXiv:2506.21563v1, CC BY 4.0. Scores extracted from Tables 2-4. (licence)