Skip to content

Benchmark

pfgen-bench Japanese generation: Helpfulness

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
2.07

Score distribution

0.219
Mean
0.132
Median
0.226
Std dev
0.014
10th percentile
0.611
90th percentile
0
Min
00.909
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.07 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Qwen3 8Bqa/chat/667d843· pfgen-bench (Preferred Networks) data docs0.253
52Qwen2.5 32Bcompletion/b356bba· pfgen-bench (Preferred Networks) data docs0.25
53Mistral Small 24B Instructqa/chat/ecbf52c· pfgen-bench (Preferred Networks) data docs0.248
54GPT-3.5 Turboqa/chat/eaeade4· pfgen-bench (Preferred Networks) data docs0.239
55Llama 3 70B Instructcompletion/8dfb479· pfgen-bench (Preferred Networks) data docs0.236
56Mistral Small 24B Instructqa/chat/f9649b1· pfgen-bench (Preferred Networks) data docs0.235
57Phi-3 Medium 4K Instructcompletion/74cf0cd· pfgen-bench (Preferred Networks) data docs0.234
58Mistral Small 24B Instructcompletion/0b0698f· pfgen-bench (Preferred Networks) data docs0.232
59Qwen3 8B Basecompletion/57fa665· pfgen-bench (Preferred Networks) data docs0.231
60Phi-3 Medium 128K Instructcompletion/3c9dc10· pfgen-bench (Preferred Networks) data docs0.223
61Llama 3.1 70B Instructcompletion/12b7638· pfgen-bench (Preferred Networks) data docs0.222
62Gemma 2 27B ITcompletion/a3fa605· pfgen-bench (Preferred Networks) data docs0.218
63Llama 3.1 70B Instructqa/chat/011aba5· pfgen-bench (Preferred Networks) data docs0.218
64Meta Llama 3 70bcompletion/a1448c0· pfgen-bench (Preferred Networks) data docs0.213
65Llama 3.1 70Bcompletion/19cc458· pfgen-bench (Preferred Networks) data docs0.211
66Gemma 2 9B ITcompletion/d29837e· pfgen-bench (Preferred Networks) data docs0.209
67Qwen1.5-72B-Chatqa/chat/d1092d0· pfgen-bench (Preferred Networks) data docs0.208
68Qwen1.5-72B-Chatcompletion/088d789· pfgen-bench (Preferred Networks) data docs0.206
69Phi-3 Medium 128K Instructqa/chat/016fa5e· pfgen-bench (Preferred Networks) data docs0.193
70Qwen2.5 14Bcompletion/174d4f0· pfgen-bench (Preferred Networks) data docs0.188
71Gemma 2 27B ITqa/chat/3329a88· pfgen-bench (Preferred Networks) data docs0.186
72Qwen2 57B A14B Instructqa/chat/d1e4d61· pfgen-bench (Preferred Networks) data docs0.173
73Qwen1.5-72Bcompletion/a4094f9· pfgen-bench (Preferred Networks) data docs0.153
74Qwen3 4B Basecompletion/af29cd4· pfgen-bench (Preferred Networks) data docs0.153
75gemma-2-9bcompletion/161d30c· pfgen-bench (Preferred Networks) data docs0.148
76DeepSeek LLM 67B Chatqa/chat/e1b18f0· pfgen-bench (Preferred Networks) data docs0.139
77Qwen1.5-32B-Chatcompletion/146f406· pfgen-bench (Preferred Networks) data docs0.133
78Qwen2.5 7Bcompletion/4d123a9· pfgen-bench (Preferred Networks) data docs0.132
79Qwen2.5 7B Instructqa/chat/81e4cf3· pfgen-bench (Preferred Networks) data docs0.132
80Qwen1.5-32B-Chatqa/chat/9df40bc· pfgen-bench (Preferred Networks) data docs0.13
81Qwen2 57B A14B Instructcompletion/bafc717· pfgen-bench (Preferred Networks) data docs0.124
82Mistral Nemocompletion/e27a5da· pfgen-bench (Preferred Networks) data docs0.122
83GLM 4 9B 0414completion/77a535b· pfgen-bench (Preferred Networks) data docs0.118
84Qwen2 7B Instructqa/chat/443a1af· pfgen-bench (Preferred Networks) data docs0.111
85Gemma 2 9B ITqa/chat/37ba4f6· pfgen-bench (Preferred Networks) data docs0.104
86Yi 1.5 34B Chatqa/chat/868c125· pfgen-bench (Preferred Networks) data docs0.101
87Mistral Nemoqa/chat/ab861fe· pfgen-bench (Preferred Networks) data docs0.1
88Llama-2-70b-chat-hfcompletion/152cf31· pfgen-bench (Preferred Networks) data docs0.099
89Qwen2 7B Instructcompletion/880bb2d· pfgen-bench (Preferred Networks) data docs0.098
90Qwen2.5 7B Instructqa/chat/20023d7· pfgen-bench (Preferred Networks) data docs0.098
91Qwen-14B-Chatcompletion/bad8c09· pfgen-bench (Preferred Networks) data docs0.097
92Qwen 2 (7B)completion/2fd6882· pfgen-bench (Preferred Networks) data docs0.093
93Qwen-14B-Chatqa/chat/7fee9c7· pfgen-bench (Preferred Networks) data docs0.092
94Llama 3 8B Instructqa/chat/3d447ac· pfgen-bench (Preferred Networks) data docs0.09
95gemma2-27bcompletion/f9b6424· pfgen-bench (Preferred Networks) data docs0.087
96gemma-7bcompletion/ebc12f4· pfgen-bench (Preferred Networks) data docs0.085
97mixtral-8x22b-v0.1completion/40ef626· pfgen-bench (Preferred Networks) data docs0.084
98Llama 3 8B Instructcompletion/93af24a· pfgen-bench (Preferred Networks) data docs0.082
99Qwen3 1.7B Basecompletion/26f4649· pfgen-bench (Preferred Networks) data docs0.081
100Qwen1.5-14B-Chatqa/chat/577f2a5· pfgen-bench (Preferred Networks) data docs0.08

Showing 51–100 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)