Skip to content

Benchmark

pfgen-bench Japanese generation: Helpfulness

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
2.07

Score distribution

0.219
Mean
0.132
Median
0.226
Std dev
0.014
10th percentile
0.611
90th percentile
0
Min
00.909
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.07 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
101Gemma 2 2B ITcompletion/a5a3763· pfgen-bench (Preferred Networks) data docs0.079
102Qwen1.5-14B-Chatcompletion/043470d· pfgen-bench (Preferred Networks) data docs0.075
103Llama 3.1 8B Instructqa/chat/e3372e7· pfgen-bench (Preferred Networks) data docs0.074
104Phi-3 Small 8K Instructqa/chat/9a69f41· pfgen-bench (Preferred Networks) data docs0.073
105Gemma 1.1 7B ITcompletion/edc5e6e· pfgen-bench (Preferred Networks) data docs0.072
106Llama 3.1 8Bcompletion/1568d43· pfgen-bench (Preferred Networks) data docs0.068
107Meta Llama 3 8bcompletion/0f9faf5· pfgen-bench (Preferred Networks) data docs0.068
108Phi-3 Small 8K Instructcompletion/02b97c5· pfgen-bench (Preferred Networks) data docs0.068
109Gemma 1.1 7B ITqa/chat/7cedd4e· pfgen-bench (Preferred Networks) data docs0.066
110Gemma 2 2B ITqa/chat/3d0cfea· pfgen-bench (Preferred Networks) data docs0.066
111Qwen1.5-14Bcompletion/4e006c4· pfgen-bench (Preferred Networks) data docs0.066
112Qwen1.5-7B-Chatqa/chat/f4731ca· pfgen-bench (Preferred Networks) data docs0.056
113Llama 3.1 8B Instructcompletion/0db64de· pfgen-bench (Preferred Networks) data docs0.055
114Mixtral 8x7B Instruct v0.1completion/024664d· pfgen-bench (Preferred Networks) data docs0.05
115gemma2-2bcompletion/4174ff9· pfgen-bench (Preferred Networks) data docs0.046
116Qwen1.5-7Bcompletion/e51e217· pfgen-bench (Preferred Networks) data docs0.046
117Llama-2-70b-chat-hfqa/chat/fddf689· pfgen-bench (Preferred Networks) data docs0.045
118Qwen1.5-7B-Chatcompletion/b75f3a7· pfgen-bench (Preferred Networks) data docs0.045
119Mixtral 8x7B Instruct v0.1qa/chat/00e933e· pfgen-bench (Preferred Networks) data docs0.043
120Mixtral 8X7b V0.1completion/077c688· pfgen-bench (Preferred Networks) data docs0.037
121Llama-2-13b-chat-hfqa/chat/7db3434· pfgen-bench (Preferred Networks) data docs0.032
122Qwen2.5 1.5Bcompletion/35940e1· pfgen-bench (Preferred Networks) data docs0.032
123Llama-2-7b-chat-hfcompletion/d7e19ff· pfgen-bench (Preferred Networks) data docs0.031
124gemma-2bcompletion/078c978· pfgen-bench (Preferred Networks) data docs0.026
125Gemma 1.1 2B ITcompletion/931162e· pfgen-bench (Preferred Networks) data docs0.025
126Qwen2 1.5b Instructqa/chat/2419c89· pfgen-bench (Preferred Networks) data docs0.025
127Gemma 2B ITcompletion/652b56f· pfgen-bench (Preferred Networks) data docs0.024
128Mistral 7B Instruct v0.2completion/1aa2ad7· pfgen-bench (Preferred Networks) data docs0.024
129Gemma 7B ITcompletion/a1c5bf9· pfgen-bench (Preferred Networks) data docs0.022
130Qwen2 1.5b Instructcompletion/b8f3248· pfgen-bench (Preferred Networks) data docs0.022
131Qwen3 0.6B Basecompletion/f504bef· pfgen-bench (Preferred Networks) data docs0.022
132Qwen1.5-4B-Chatqa/chat/49c9c69· pfgen-bench (Preferred Networks) data docs0.021
133Gemma 7B ITqa/chat/a29928b· pfgen-bench (Preferred Networks) data docs0.02
134Mistral 7B Instruct v0.2qa/chat/a690ecc· pfgen-bench (Preferred Networks) data docs0.019
135Qwen 2 (1.5B)completion/a177c05· pfgen-bench (Preferred Networks) data docs0.019
136Qwen1.5-4B-Chatcompletion/843369c· pfgen-bench (Preferred Networks) data docs0.019
137Mistral 7B v0.1completion/7f376df· pfgen-bench (Preferred Networks) data docs0.018
138Qwen1.5-1.8B-Chatcompletion/c0f1d5a· pfgen-bench (Preferred Networks) data docs0.017
139Qwen1.5-4Bcompletion/681d1a9· pfgen-bench (Preferred Networks) data docs0.015
140Mistral (7B) Instruct v0.1completion/01c558f· pfgen-bench (Preferred Networks) data docs0.014
141Qwen1.5-1.8Bcompletion/7343413· pfgen-bench (Preferred Networks) data docs0.014
142Mistral (7B) Instruct v0.1qa/chat/1be1e40· pfgen-bench (Preferred Networks) data docs0.012
143DeepSeek LLM 67B Chatcompletion/d0ebf05· pfgen-bench (Preferred Networks) data docs0.009
144Llama-2-7b-chat-hfqa/chat/18e34ca· pfgen-bench (Preferred Networks) data docs0.009
145Qwen1.5-0.5B-Chatcompletion/4f5c753· pfgen-bench (Preferred Networks) data docs0.009
146qwen2-0.5b-instructcompletion/24880e5· pfgen-bench (Preferred Networks) data docs0.007
147qwen2-0.5b-instructqa/chat/34dccbe· pfgen-bench (Preferred Networks) data docs0.006
148Yi 1.5 34B Chatcompletion/39ffa65· pfgen-bench (Preferred Networks) data docs0.006
149Gemma 1.1 2B ITqa/chat/ef0d7c6· pfgen-bench (Preferred Networks) data docs0.003
150Qwen1.5-0.5Bcompletion/0e72cbc· pfgen-bench (Preferred Networks) data docs0.003

Showing 101–150 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)