Skip to content

Benchmark

Benchmark Nusantara jv-A pilihan-ganda (prompt mcq-py1, scorer mcq-py1, items c0c83850858e, key 18e8ec378904, withheld slice)

Multilingual benchmarks compare model performance across languages. Higher scores are better.

Models measured
18
Scores recorded
18
Sources
1
IQ weight
1.12

Score distribution

0.808
Mean
0.909
Median
0.182
Std dev
0.546
10th percentile
1
90th percentile
0.455
Min
0.4551
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.12 for chat models, measured over 18 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 5.5repetition: 1, route anthropic/claude-sonnet-5.5, token cap 2400· Benchmark Nusantara (ibahasa) data docs1
2DeepSeek V4 Flash 0731repetition: 1, route deepseek/deepseek-v4-flash-0731, token cap 2400· Benchmark Nusantara (ibahasa) data docs1
3DeepSeek V4.1 Flashrepetition: 1, route deepseek/deepseek-v4.1-flash, token cap 2400· Benchmark Nusantara (ibahasa) data docs1
4Gemini 3.5 Flashrepetition: 1, route google/gemini-3.5-flash, token cap 600· Benchmark Nusantara (ibahasa) data docs1
5GPT-5.6 Lunarepetition: 1, route openai/gpt-5.6-luna, token cap 600· Benchmark Nusantara (ibahasa) data docs1
6GPT-6 Astrarepetition: 1, route openai/gpt-6-astra, token cap 2400· Benchmark Nusantara (ibahasa) data docs1
7Claude Haiku 5.5repetition: 1, route anthropic/claude-haiku-5.5, token cap 2400· Benchmark Nusantara (ibahasa) data docs0.909
8Gemma 4 31B ITrepetition: 1, route google/gemma-4-31b-it, token cap 600· Benchmark Nusantara (ibahasa) data docs0.909
9GLM 5.3 Flashrepetition: 1, route z-ai/glm-5.3-flash, token cap 9600· Benchmark Nusantara (ibahasa) data docs0.909
10MiMo V2.6 Flashrepetition: 1, route xiaomi/mimo-v2.6-flash, token cap 9600· Benchmark Nusantara (ibahasa) data docs0.818
11Claude Haiku 4.5repetition: 1, route anthropic/claude-haiku-4.5, token cap 600· Benchmark Nusantara (ibahasa) data docs0.727
12GLM 5.3repetition: 1, route z-ai/glm-5.3, token cap 2400· Benchmark Nusantara (ibahasa) data docs0.727
13Ling 3.0 Flashrepetition: 1, route inclusionai/ling-3.0-flash, token cap 2400· Benchmark Nusantara (ibahasa) data docs0.727
14Mistral Small 24B Instructrepetition: 1, route mistralai/mistral-small-24b-instruct-2501, token cap 600, 2501· Benchmark Nusantara (ibahasa) data docs0.636
15Seed 2.0 Minirepetition: 1, route bytedance-seed/seed-2.0-mini, token cap 12000· Benchmark Nusantara (ibahasa) data docs0.636
16Llama 3.1 8B Instructrepetition: 1, local Q4_K_M, token cap 600· Benchmark Nusantara (ibahasa) data docs0.546
17Solar Pro 4repetition: 1, route upstage/solar-pro4, token cap 600· Benchmark Nusantara (ibahasa) data docs0.546
18Qwen3 30B A3B Instructrepetition: 1, route qwen/qwen3-30b-a3b-instruct-2507, token cap 600· Benchmark Nusantara (ibahasa) data docs0.455

Data credits: Benchmark Nusantara, ibahasa (M. Khoirul Huda), doi:10.5281/zenodo.21970463. Test results CC BY 4.0, attribution to ibahasa; aggregate results CSV only. (licence)