Skip to content

Benchmark

pfgen-bench Japanese generation: Helpfulness

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
2.07

Score distribution

0.219
Mean
0.132
Median
0.226
Std dev
0.014
10th percentile
0.611
90th percentile
0
Min
00.909
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.07 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 4qa/chat/2dafa10· pfgen-bench (Preferred Networks) data docs0.909
2Claude Opus 4qa/chat/d33465d· pfgen-bench (Preferred Networks) data docs0.899
3Claude 3.5 Sonnetqa/chat/2938dda· pfgen-bench (Preferred Networks) data docs0.883
4ChatGPT-4o Latestqa/chat/c275a7b· pfgen-bench (Preferred Networks) data docs0.87
5DeepSeek V3completion/4a38349· pfgen-bench (Preferred Networks) data docs0.822
6DeepSeek R1qa/chat/5abaa36· pfgen-bench (Preferred Networks) data docs0.807
7ChatGPT-4o Latestqa/chat/f029db6· pfgen-bench (Preferred Networks) data docs0.74
8GPT 4oqa/chat/5e1e9f4· pfgen-bench (Preferred Networks) data docs0.686
9Qwen Maxqa/chat/0944f44· pfgen-bench (Preferred Networks) data docs0.679
10Qwen Maxqa/chat/fa9558d· pfgen-bench (Preferred Networks) data docs0.676
11gemini-1.5-pro-002qa/chat/873c5b2· pfgen-bench (Preferred Networks) data docs0.656
12MiniMax Text 01qa/chat/74d4af0· pfgen-bench (Preferred Networks) data docs0.638
13GPT-4 Turboqa/chat/e96542c· pfgen-bench (Preferred Networks) data docs0.632
14ChatGPT-4o Latestqa/chat/84cddfa· pfgen-bench (Preferred Networks) data docs0.631
15Claude 3 Opusqa/chat/89cdaa9· pfgen-bench (Preferred Networks) data docs0.627
16MiniMax Text 01qa/chat/05c1dce· pfgen-bench (Preferred Networks) data docs0.611
17Qwen 3 235B A22Bcompletion/89946a4· pfgen-bench (Preferred Networks) data docs0.585
18Qwen2.5 72B Instructqa/chat/5705734· pfgen-bench (Preferred Networks) data docs0.549
19Qwen2.5 72B Instructqa/chat/5e498db· pfgen-bench (Preferred Networks) data docs0.54
20GPT 4qa/chat/8cad1b9· pfgen-bench (Preferred Networks) data docs0.536
21gemini-1.5-flash-002qa/chat/b868ea0· pfgen-bench (Preferred Networks) data docs0.519
22Claude 3 Sonnetqa/chat/f70c14d· pfgen-bench (Preferred Networks) data docs0.476
23Gemini 2.0 Flash Experimentalqa/chat/5f7c6a9· pfgen-bench (Preferred Networks) data docs0.476
24Mistral Large Instruct 2411qa/chat/c6a3714· pfgen-bench (Preferred Networks) data docs0.469
25Mistral Large Instruct 2411qa/chat/564af5f· pfgen-bench (Preferred Networks) data docs0.458
26Nemotron-4 340B Instructqa/chat/0d4b478· pfgen-bench (Preferred Networks) data docs0.435
27Nemotron-4 340B Instructcompletion/e99a1d3· pfgen-bench (Preferred Networks) data docs0.429
28Mistral Large Instruct 2411completion/b2f50ab· pfgen-bench (Preferred Networks) data docs0.415
29Qwen2 72B Instructqa/chat/baf6c82· pfgen-bench (Preferred Networks) data docs0.386
30Qwen3 32Bcompletion/9f132a0· pfgen-bench (Preferred Networks) data docs0.378
31Llama 3.1 405Bcompletion/d0c2319· pfgen-bench (Preferred Networks) data docs0.368
32Qwen2.5 32B Instructqa/chat/01f44c1· pfgen-bench (Preferred Networks) data docs0.36
33Gemini 2.0 Flash Experimentalqa/chat/c246287, thinking· pfgen-bench (Preferred Networks) data docs0.354
34Qwen2.5 32B Instructqa/chat/8a041a4· pfgen-bench (Preferred Networks) data docs0.349
35Phi 4qa/chat/af856d5· pfgen-bench (Preferred Networks) data docs0.34
36Phi 4qa/chat/772e05d· pfgen-bench (Preferred Networks) data docs0.337
37Qwen2.5 72Bcompletion/8c7477d· pfgen-bench (Preferred Networks) data docs0.337
38Phi 4completion/118d26c· pfgen-bench (Preferred Networks) data docs0.309
39Qwen 2.5 14B Instructqa/chat/54fea07· pfgen-bench (Preferred Networks) data docs0.308
40QwQ 32B Previewqa/chat/a240792· pfgen-bench (Preferred Networks) data docs0.301
41Qwen2 72B Instructcompletion/c1ddc4d· pfgen-bench (Preferred Networks) data docs0.299
42Claude 3 Haikuqa/chat/9e7a056· pfgen-bench (Preferred Networks) data docs0.298
43QwQ 32B Previewqa/chat/a362164· pfgen-bench (Preferred Networks) data docs0.294
44Qwen3 30B A3b Basecompletion/8d6c0f4· pfgen-bench (Preferred Networks) data docs0.292
45Qwen 2.5 14B Instructqa/chat/4a65a55· pfgen-bench (Preferred Networks) data docs0.29
46Qwen 2 (72B)completion/ff6791f· pfgen-bench (Preferred Networks) data docs0.279
47Qwen3 8Bcompletion/9d7e974· pfgen-bench (Preferred Networks) data docs0.278
48Qwen3 14B Basecompletion/3e8d9d4· pfgen-bench (Preferred Networks) data docs0.255
49Llama 3 70B Instructqa/chat/3c91259· pfgen-bench (Preferred Networks) data docs0.254
50Phi-3 Medium 4K Instructqa/chat/d396e2e· pfgen-bench (Preferred Networks) data docs0.253

Showing 1–50 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)