Skip to content

Benchmark

pfgen-bench Japanese generation: Truthfulness

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
1.36

Score distribution

0.7
Mean
0.734
Median
0.215
Std dev
0.392
10th percentile
0.959
90th percentile
0.001
Min
0.0010.988
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.36 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
101gemma-7bcompletion/ebc12f4· pfgen-bench (Preferred Networks) data docs0.631
102Phi-3 Small 8K Instructqa/chat/9a69f41· pfgen-bench (Preferred Networks) data docs0.63
103Yi 1.5 34B Chatqa/chat/868c125· pfgen-bench (Preferred Networks) data docs0.628
104Gemma 1.1 2B ITcompletion/931162e· pfgen-bench (Preferred Networks) data docs0.625
105Llama 3.1 8B Instructqa/chat/e3372e7· pfgen-bench (Preferred Networks) data docs0.624
106Qwen1.5-32B-Chatqa/chat/9df40bc· pfgen-bench (Preferred Networks) data docs0.62
107gemma2-2bcompletion/4174ff9· pfgen-bench (Preferred Networks) data docs0.616
108mixtral-8x22b-v0.1completion/40ef626· pfgen-bench (Preferred Networks) data docs0.615
109Llama 3 8B Instructqa/chat/3d447ac· pfgen-bench (Preferred Networks) data docs0.614
110Mistral Nemocompletion/e27a5da· pfgen-bench (Preferred Networks) data docs0.611
111Qwen1.5-14Bcompletion/4e006c4· pfgen-bench (Preferred Networks) data docs0.609
112gemma2-27bcompletion/f9b6424· pfgen-bench (Preferred Networks) data docs0.608
113Llama-2-70b-chat-hfcompletion/152cf31· pfgen-bench (Preferred Networks) data docs0.608
114Qwen1.5-14B-Chatqa/chat/577f2a5· pfgen-bench (Preferred Networks) data docs0.604
115Qwen1.5-14B-Chatcompletion/043470d· pfgen-bench (Preferred Networks) data docs0.593
116Llama 3.1 8B Instructcompletion/0db64de· pfgen-bench (Preferred Networks) data docs0.589
117Gemma 2B ITcompletion/652b56f· pfgen-bench (Preferred Networks) data docs0.577
118Mixtral 8X7b V0.1completion/077c688· pfgen-bench (Preferred Networks) data docs0.573
119Llama-2-7b-chat-hfcompletion/d7e19ff· pfgen-bench (Preferred Networks) data docs0.57
120Mistral Nemoqa/chat/ab861fe· pfgen-bench (Preferred Networks) data docs0.563
121Mixtral 8x7B Instruct v0.1qa/chat/00e933e· pfgen-bench (Preferred Networks) data docs0.548
122Qwen2.5 1.5Bcompletion/35940e1· pfgen-bench (Preferred Networks) data docs0.533
123Qwen1.5-7B-Chatcompletion/b75f3a7· pfgen-bench (Preferred Networks) data docs0.52
124Qwen2 1.5b Instructcompletion/b8f3248· pfgen-bench (Preferred Networks) data docs0.513
125Qwen 2 (1.5B)completion/a177c05· pfgen-bench (Preferred Networks) data docs0.511
126Qwen2 1.5b Instructqa/chat/2419c89· pfgen-bench (Preferred Networks) data docs0.511
127Qwen1.5-7Bcompletion/e51e217· pfgen-bench (Preferred Networks) data docs0.504
128Qwen1.5-7B-Chatqa/chat/f4731ca· pfgen-bench (Preferred Networks) data docs0.503
129Qwen3 0.6B Basecompletion/f504bef· pfgen-bench (Preferred Networks) data docs0.492
130gemma-2bcompletion/078c978· pfgen-bench (Preferred Networks) data docs0.48
131Mistral 7B v0.1completion/7f376df· pfgen-bench (Preferred Networks) data docs0.454
132Llama-2-13b-chat-hfqa/chat/7db3434· pfgen-bench (Preferred Networks) data docs0.453
133Qwen1.5-1.8B-Chatcompletion/c0f1d5a· pfgen-bench (Preferred Networks) data docs0.445
134Qwen1.5-4B-Chatcompletion/843369c· pfgen-bench (Preferred Networks) data docs0.441
135DeepSeek LLM 67B Chatcompletion/d0ebf05· pfgen-bench (Preferred Networks) data docs0.44
136qwen2-0.5b-instructcompletion/24880e5· pfgen-bench (Preferred Networks) data docs0.426
137Qwen1.5-1.8Bcompletion/7343413· pfgen-bench (Preferred Networks) data docs0.424
138Mistral 7B Instruct v0.2completion/1aa2ad7· pfgen-bench (Preferred Networks) data docs0.419
139Mistral (7B) Instruct v0.1completion/01c558f· pfgen-bench (Preferred Networks) data docs0.4
140Qwen1.5-1.8B-Chatqa/chat/a452608· pfgen-bench (Preferred Networks) data docs0.392
141Qwen1.5-4B-Chatqa/chat/49c9c69· pfgen-bench (Preferred Networks) data docs0.39
142Qwen1.5-4Bcompletion/681d1a9· pfgen-bench (Preferred Networks) data docs0.388
143qwen2-0.5b-instructqa/chat/34dccbe· pfgen-bench (Preferred Networks) data docs0.379
144Gemma 1.1 2B ITqa/chat/ef0d7c6· pfgen-bench (Preferred Networks) data docs0.376
145Mistral (7B) Instruct v0.1qa/chat/1be1e40· pfgen-bench (Preferred Networks) data docs0.362
146phi-2completion/c8c91cf· pfgen-bench (Preferred Networks) data docs0.356
147Mistral 7B Instruct v0.2qa/chat/a690ecc· pfgen-bench (Preferred Networks) data docs0.334
148Qwen1.5-0.5B-Chatcompletion/4f5c753· pfgen-bench (Preferred Networks) data docs0.327
149Qwen1.5-0.5Bcompletion/0e72cbc· pfgen-bench (Preferred Networks) data docs0.314
150Llama-2-70b-chat-hfqa/chat/fddf689· pfgen-bench (Preferred Networks) data docs0.303

Showing 101–150 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)