Skip to content

Benchmark

Swallow v2 Japanese post-trained: MMLU-ProX

Higher scores are better.

Models measured
41
Scores recorded
41
Sources
1
IQ weight
2.25

Score distribution

0.668
Mean
0.741
Median
0.184
Std dev
0.335
10th percentile
0.835
90th percentile
0.148
Min
0.1480.852
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.25 for chat models, measured over 41 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5.4reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.852
2GPT 5reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.849
3Gemma 4 31B ITreasoning on· Swallow LLM Leaderboard v2 data docs0.841
4Qwen3.5 122B-A10Breasoning on· Swallow LLM Leaderboard v2 data docs0.841
5OpenAI o3reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.835
6Qwen3.5 27Breasoning on· Swallow LLM Leaderboard v2 data docs0.835
7Qwen3.5 35B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.829
8Qwen 3 235B A22B Thinking 2507reasoning on· Swallow LLM Leaderboard v2 data docs0.819
9Gemma 4 26B A4Breasoning on· Swallow LLM Leaderboard v2 data docs0.818
10GPT-5 minireasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.805
11Qwen3 235B A22B Instruct 2507· Swallow LLM Leaderboard v2 data docs0.799
12Qwen3 Next 80B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.797
13Nemotron 3 Super 120B A12Breasoning on· Swallow LLM Leaderboard v2 data docs0.794
14Qwen3.5 9Breasoning on· Swallow LLM Leaderboard v2 data docs0.784
15GPT 4.1· Swallow LLM Leaderboard v2 data docs0.772
16Qwen3 Next 80B A3B Instruct· Swallow LLM Leaderboard v2 data docs0.77
17o3 Minireasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.76
18Qwen3 32Breasoning on· Swallow LLM Leaderboard v2 data docs0.759
19GPT OSS 120Breasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.754
20Qwen3.5 4Breasoning on· Swallow LLM Leaderboard v2 data docs0.75
21Qwen3 30B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.741
22Qwen3 14Breasoning on· Swallow LLM Leaderboard v2 data docs0.74
23Qwen3 8Breasoning on· Swallow LLM Leaderboard v2 data docs0.711
24GPT OSS 20breasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.705
25Llama 4 Scout· Swallow LLM Leaderboard v2 data docs0.687
26GPT 4o· Swallow LLM Leaderboard v2 data docs0.685
27Gemma 4 E4B-itreasoning on· Swallow LLM Leaderboard v2 data docs0.67
28Qwen3 4Breasoning on· Swallow LLM Leaderboard v2 data docs0.665
29Olmo 3 32B Thinkreasoning on· Swallow LLM Leaderboard v2 data docs0.663
30Gemma 3 27B IT· Swallow LLM Leaderboard v2 data docs0.609
31Llama 3.3 70B Instruct· Swallow LLM Leaderboard v2 data docs0.607
32Gemma 4 E2B-itreasoning on· Swallow LLM Leaderboard v2 data docs0.58
33Gemma 3 12B IT· Swallow LLM Leaderboard v2 data docs0.527
34Qwen3 1.7Breasoning on· Swallow LLM Leaderboard v2 data docs0.522
35Olmo 3 7B Thinkreasoning on· Swallow LLM Leaderboard v2 data docs0.491
36Qwen3.5 2Breasoning on· Swallow LLM Leaderboard v2 data docs0.391
37Gemma 3 4B IT· Swallow LLM Leaderboard v2 data docs0.335
38Qwen3 0.6Breasoning on· Swallow LLM Leaderboard v2 data docs0.32
39Llama 3.1 8B Instruct· Swallow LLM Leaderboard v2 data docs0.306
40Qwen3.5 0.8Breasoning on· Swallow LLM Leaderboard v2 data docs0.236
41Gemma 3 1b it· Swallow LLM Leaderboard v2 data docs0.148

Data credits: Swallow LLM Leaderboard v2, Swallow LLM team (Institute of Science Tokyo). Contents CC BY 4.0; Japanese post-trained task scores on the source's 0..1 scale. (licence)