Skip to content

Benchmark

FormosanBench MT BLEU (Mandarin→Atayal, zero-shot)

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models measured
5
Scores recorded
5
Sources
1
IQ weight
0.00

Score distribution

0
Mean
0.001
Median
0
Std dev
0
10th percentile
0.001
90th percentile
0.0001
Min

This benchmark has a weight of 0.00 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Mistral (7B) Instruct v0.3zero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.001
2Llama 3.1 70Bzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0006
3Gemma 1.1 7B ITzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0005
4GPT-4o minizero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0001
5Llama 3.1 8Bzero-shot, Mandarin prompt· FormosanBench (arXiv:2506.21563v1) data0.0001

Data credits: Lin, Chen and Zhang, FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models, arXiv:2506.21563v1, CC BY 4.0. Scores extracted from Tables 2-4. (licence)