Skip to content

Benchmark

Swallow v2 Japanese post-trained: JHumanEval

Coding benchmarks compare models on programming tasks. Higher scores are better.

Models measured
41
Scores recorded
41
Sources
1
IQ weight
2.50

Score distribution

0.803
Mean
0.905
Median
0.239
Std dev
0.58
10th percentile
0.946
90th percentile
0.012
Min
0.0120.963
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.50 for chat models, measured over 41 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1GPT 5.4reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.963
2Gemma 4 26B A4Breasoning on· Swallow LLM Leaderboard v2 data docs0.959
3Gemma 4 31B ITreasoning on· Swallow LLM Leaderboard v2 data docs0.952
4Qwen 3 235B A22B Thinking 2507reasoning on· Swallow LLM Leaderboard v2 data docs0.948
5GPT 5reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.946
6GPT-5 minireasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.944
7OpenAI o3reasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.937
8Qwen3.5 122B-A10Breasoning on· Swallow LLM Leaderboard v2 data docs0.937
9GPT OSS 20breasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.932
10Nemotron 3 Super 120B A12Breasoning on· Swallow LLM Leaderboard v2 data docs0.93
11Qwen3.5 27Breasoning on· Swallow LLM Leaderboard v2 data docs0.93
12o3 Minireasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.928
13GPT OSS 120Breasoning on (medium)· Swallow LLM Leaderboard v2 data docs0.927
14Olmo 3 32B Thinkreasoning on· Swallow LLM Leaderboard v2 data docs0.921
15Qwen3 14Breasoning on· Swallow LLM Leaderboard v2 data docs0.921
16Qwen3.5 35B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.921
17Qwen3 30B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.915
18Qwen3 32Breasoning on· Swallow LLM Leaderboard v2 data docs0.915
19GPT 4.1· Swallow LLM Leaderboard v2 data docs0.911
20Qwen3.5 9Breasoning on· Swallow LLM Leaderboard v2 data docs0.909
21Qwen3 Next 80B A3B Instruct· Swallow LLM Leaderboard v2 data docs0.905
22Qwen3 235B A22B Instruct 2507· Swallow LLM Leaderboard v2 data docs0.9
23Olmo 3 7B Thinkreasoning on· Swallow LLM Leaderboard v2 data docs0.896
24Qwen3 8Breasoning on· Swallow LLM Leaderboard v2 data docs0.891
25Gemma 4 E4B-itreasoning on· Swallow LLM Leaderboard v2 data docs0.882
26Qwen3 4Breasoning on· Swallow LLM Leaderboard v2 data docs0.879
27Qwen3 Next 80B A3Breasoning on· Swallow LLM Leaderboard v2 data docs0.873
28GPT 4o· Swallow LLM Leaderboard v2 data docs0.844
29Gemma 4 E2B-itreasoning on· Swallow LLM Leaderboard v2 data docs0.826
30Llama 4 Scout· Swallow LLM Leaderboard v2 data docs0.82
31Gemma 3 27B IT· Swallow LLM Leaderboard v2 data docs0.796
32Gemma 3 12B IT· Swallow LLM Leaderboard v2 data docs0.763
33Qwen3 1.7Breasoning on· Swallow LLM Leaderboard v2 data docs0.757
34Llama 3.3 70B Instruct· Swallow LLM Leaderboard v2 data docs0.752
35Qwen3.5 4Breasoning on· Swallow LLM Leaderboard v2 data docs0.727
36Gemma 3 4B IT· Swallow LLM Leaderboard v2 data docs0.604
37Llama 3.1 8B Instruct· Swallow LLM Leaderboard v2 data docs0.58
38Qwen3 0.6Breasoning on· Swallow LLM Leaderboard v2 data docs0.446
39Gemma 3 1b it· Swallow LLM Leaderboard v2 data docs0.112
40Qwen3.5 0.8Breasoning on· Swallow LLM Leaderboard v2 data docs0.013
41Qwen3.5 2Breasoning on· Swallow LLM Leaderboard v2 data docs0.012

Data credits: Swallow LLM Leaderboard v2, Swallow LLM team (Institute of Science Tokyo). Contents CC BY 4.0; Japanese post-trained task scores on the source's 0..1 scale. (licence)