Skip to content

Benchmark

Swallow v2 Japanese post-trained: MIFEvalJa

Instruction following benchmarks measure how closely models follow instructions. Higher scores are better.

Models measured
41
Scores recorded
41
Sources
1
IQ weight
1.08

Score distribution

0.639
Mean
0.65
Median
0.179
Std dev
0.345
10th percentile
0.85
90th percentile
0.261
Min
0.2610.907
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.08 for chat models, measured over 41 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.907
2Gemma 4 31B ITreasoning on· Swallow LLM Leaderboard v2 data docs0.898
3GPT 5.4reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.881
4Gemma 4 26B A4Breasoning on· Swallow LLM Leaderboard v2 data docs0.867
5OpenAI o3reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.85
6o3 Minireasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.841
7GPT-5 minireasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.827
8Nemotron 3 Super 120B A12Breasoning on· Swallow LLM Leaderboard v2 data docs0.823
9Qwen3 Next 80B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.819
10GPT 4.1· Swallow LLM Leaderboard v2 data docs0.81
11Qwen 3 235B A22B Thinking 2507reasoning on· Swallow LLM Leaderboard v2 data docs0.81
12Qwen3.5 122B-A10Breasoning on· Swallow LLM Leaderboard v2 data docs0.774
13GPT OSS 120Breasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.735
14Qwen3.5 27Breasoning on· Swallow LLM Leaderboard v2 data docs0.735
15Qwen3 235B A22B Instruct 2507· Swallow LLM Leaderboard v2 data docs0.73
16GPT OSS 20breasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.726
17GPT 4o· Swallow LLM Leaderboard v2 data docs0.704
18Qwen3 Next 80B A3B Instruct· Swallow LLM Leaderboard v2 data docs0.681
19Qwen3 30B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.664
20Qwen3.5 35B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.659
21Llama 3.3 70B Instruct· Swallow LLM Leaderboard v2 data docs0.65
22Gemma 4 E2B-itreasoning on· Swallow LLM Leaderboard v2 data docs0.646
23Gemma 4 E4B-itreasoning on· Swallow LLM Leaderboard v2 data docs0.637
24Qwen3 32Breasoning on· Swallow LLM Leaderboard v2 data docs0.637
25Gemma 3 12B IT· Swallow LLM Leaderboard v2 data docs0.619
26Llama 4 Scout· Swallow LLM Leaderboard v2 data docs0.611
27Qwen3 14Breasoning on· Swallow LLM Leaderboard v2 data docs0.611
28Qwen3 8Breasoning on· Swallow LLM Leaderboard v2 data docs0.611
29Gemma 3 27B IT· Swallow LLM Leaderboard v2 data docs0.597
30Qwen3.5 9Breasoning on· Swallow LLM Leaderboard v2 data docs0.518
31Qwen3 4Breasoning on· Swallow LLM Leaderboard v2 data docs0.509
32Qwen3 1.7Breasoning on· Swallow LLM Leaderboard v2 data docs0.491
33Gemma 3 4B IT· Swallow LLM Leaderboard v2 data docs0.473
34Qwen3.5 4Breasoning on· Swallow LLM Leaderboard v2 data docs0.456
35Qwen3 0.6Breasoning on· Swallow LLM Leaderboard v2 data docs0.425
36Llama 3.1 8B Instruct· Swallow LLM Leaderboard v2 data docs0.381
37Olmo 3 32B Thinkreasoning on· Swallow LLM Leaderboard v2 data docs0.345
38Qwen3.5 2Breasoning on· Swallow LLM Leaderboard v2 data docs0.332
39Gemma 3 1b it· Swallow LLM Leaderboard v2 data docs0.323
40Olmo 3 7B Thinkreasoning on· Swallow LLM Leaderboard v2 data docs0.314
41Qwen3.5 0.8Breasoning on· Swallow LLM Leaderboard v2 data docs0.261

Data credits: Swallow LLM Leaderboard v2, Swallow LLM team (Institute of Science Tokyo). Contents CC BY 4.0; Japanese post-trained task scores on the source's 0..1 scale. (licence)