Benchmark
Benchmark Nusantara jv-A pilihan-ganda (prompt mcq-py1, scorer mcq-py1, items c0c83850858e, key 18e8ec378904, public slice)
Multilingual benchmarks compare model performance across languages. Higher scores are better.
- Models measured
- 18
- Scores recorded
- 18
- Sources
- 1
- IQ weight
- 1.31
Score distribution
- 0.757
- Mean
- 0.815
- Median
- 0.203
- Std dev
- 0.482
- 10th percentile
- 0.963
- 90th percentile
- 0.333
- Min
This benchmark has a weight of 1.31 for chat models, measured over 18 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Claude Sonnet 5.5repetition: 1, route anthropic/claude-sonnet-5.5, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.963 |
| 2 | DeepSeek V4.1 Flashrepetition: 1, route deepseek/deepseek-v4.1-flash, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.963 |
| 3 | Gemini 3.5 Flashrepetition: 1, route google/gemini-3.5-flash, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.963 |
| 4 | Gemma 4 31B ITrepetition: 1, route google/gemma-4-31b-it, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.963 |
| 5 | GPT-6 Astrarepetition: 1, route openai/gpt-6-astra, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.963 |
| 6 | DeepSeek V4 Flash 0731repetition: 1, route deepseek/deepseek-v4-flash-0731, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.926 |
| 7 | Claude Haiku 5.5repetition: 1, route anthropic/claude-haiku-5.5, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.889 |
| 8 | GPT-5.6 Lunarepetition: 1, route openai/gpt-5.6-luna, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.852 |
| 9 | GLM 5.3 Flashrepetition: 1, route z-ai/glm-5.3-flash, token cap 9600· Benchmark Nusantara (ibahasa) data docs | 0.815 |
| 10 | MiMo V2.6 Flashrepetition: 1, route xiaomi/mimo-v2.6-flash, token cap 9600· Benchmark Nusantara (ibahasa) data docs | 0.815 |
| 11 | GLM 5.3repetition: 1, route z-ai/glm-5.3, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.778 |
| 12 | Claude Haiku 4.5repetition: 1, route anthropic/claude-haiku-4.5, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.741 |
| 13 | Ling 3.0 Flashrepetition: 1, route inclusionai/ling-3.0-flash, token cap 2400· Benchmark Nusantara (ibahasa) data docs | 0.667 |
| 14 | Solar Pro 4repetition: 1, route upstage/solar-pro4, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.593 |
| 15 | Qwen3 30B A3B Instructrepetition: 1, route qwen/qwen3-30b-a3b-instruct-2507, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.519 |
| 16 | Seed 2.0 Minirepetition: 1, route bytedance-seed/seed-2.0-mini, token cap 12000· Benchmark Nusantara (ibahasa) data docs | 0.482 |
| 17 | Mistral Small 24B Instructrepetition: 1, route mistralai/mistral-small-24b-instruct-2501, token cap 600, 2501· Benchmark Nusantara (ibahasa) data docs | 0.407 |
| 18 | Llama 3.1 8B Instructrepetition: 1, local Q4_K_M, token cap 600· Benchmark Nusantara (ibahasa) data docs | 0.333 |
Data credits: Benchmark Nusantara, ibahasa (M. Khoirul Huda), doi:10.5281/zenodo.21970463. Test results CC BY 4.0, attribution to ibahasa; aggregate results CSV only. (licence)