Skip to content

Benchmark

CulturALL Hard

Multilingual benchmarks compare model performance across languages. Scores are percentages. Higher is better.

Models measured
8
Scores recorded
15
Sources
1
IQ weight
0.74

Score distribution

9.2%
Mean
9.3%
Median
4%
Std dev
4.7%
10th percentile
15.5%
90th percentile
4.2%
Min
4.2%18.5%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.74 for chat models, measured over 8 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 2.5 Proreasoning: auto, web: yes· CulturALL (arXiv:2604.19262v1, Table 2) data18.5%
2Gemini 2.5 Proreasoning: 128 tokens, web: yes· CulturALL (arXiv:2604.19262v1, Table 2) data15.5%
3Gemini 2.5 Flashreasoning: auto, web: yes· CulturALL (arXiv:2604.19262v1, Table 2) data12.8%
4GPT 5reasoning: high, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data10.3%
5Gemini 2.5 Proreasoning: auto, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data10.1%
6Claude Opus 4reasoning: 1024 tokens, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data9.4%
7GPT 5reasoning: low, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data9.4%
8GPT 5reasoning: medium, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data9.3%
9Claude Opus 4reasoning: disabled, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data9%
10Claude Sonnet 4reasoning: 1024 tokens, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data8.8%
11Qwen Maxreasoning: hybrid, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data5.7%
12Qwen Maxreasoning: hybrid, web: yes· CulturALL (arXiv:2604.19262v1, Table 2) data5.1%
13Qwen3 30B A3Breasoning: hybrid, web: yes· CulturALL (arXiv:2604.19262v1, Table 2) data4.7%
14Qwen 3 235B A22Breasoning: hybrid, web: no· CulturALL (arXiv:2604.19262v1, Table 2) data4.7%
15Qwen 3 235B A22Breasoning: hybrid, web: yes· CulturALL (arXiv:2604.19262v1, Table 2) data4.2%

Data credits: Lin et al., CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks, arXiv:2604.19262v1, CC BY 4.0. Table 2 scores extracted without change. (licence)