Skip to content

Benchmark

pfgen-bench Japanese generation: Fluency

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
1.56

Score distribution

0.573
Mean
0.557
Median
0.195
Std dev
0.351
10th percentile
0.839
90th percentile
0.001
Min
0.0011
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.56 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Meta Llama 3 70bcompletion/a1448c0· pfgen-bench (Preferred Networks) data docs0.664
52Qwen3 8B Basecompletion/57fa665· pfgen-bench (Preferred Networks) data docs0.661
53GPT-3.5 Turboqa/chat/eaeade4· pfgen-bench (Preferred Networks) data docs0.658
54Llama 3 70B Instructcompletion/8dfb479· pfgen-bench (Preferred Networks) data docs0.655
55Mistral Small 24B Instructqa/chat/f9649b1· pfgen-bench (Preferred Networks) data docs0.648
56Qwen2.5 14Bcompletion/174d4f0· pfgen-bench (Preferred Networks) data docs0.648
57Qwen 2.5 14B Instructqa/chat/54fea07· pfgen-bench (Preferred Networks) data docs0.644
58Mistral Small 24B Instructqa/chat/ecbf52c· pfgen-bench (Preferred Networks) data docs0.64
59Gemini 2.0 Flash Experimentalqa/chat/c246287, thinking· pfgen-bench (Preferred Networks) data docs0.639
60Qwen 2.5 14B Instructqa/chat/4a65a55· pfgen-bench (Preferred Networks) data docs0.639
61Qwen3 8Bqa/chat/667d843· pfgen-bench (Preferred Networks) data docs0.638
62Llama 3.1 70B Instructcompletion/12b7638· pfgen-bench (Preferred Networks) data docs0.636
63Qwen2.5 32B Instructqa/chat/01f44c1· pfgen-bench (Preferred Networks) data docs0.634
64Qwen2.5 32B Instructqa/chat/8a041a4· pfgen-bench (Preferred Networks) data docs0.634
65Phi-3 Medium 4K Instructqa/chat/d396e2e· pfgen-bench (Preferred Networks) data docs0.633
66Llama 3.1 70B Instructqa/chat/011aba5· pfgen-bench (Preferred Networks) data docs0.617
67Qwen2 57B A14B Instructqa/chat/d1e4d61· pfgen-bench (Preferred Networks) data docs0.615
68Qwen2 57B A14B Instructcompletion/bafc717· pfgen-bench (Preferred Networks) data docs0.601
69Gemma 2 27B ITcompletion/a3fa605· pfgen-bench (Preferred Networks) data docs0.599
70gemma-2-9bcompletion/161d30c· pfgen-bench (Preferred Networks) data docs0.595
71Qwen1.5-72Bcompletion/a4094f9· pfgen-bench (Preferred Networks) data docs0.591
72Phi-3 Medium 128K Instructqa/chat/016fa5e· pfgen-bench (Preferred Networks) data docs0.589
73Qwen1.5-72B-Chatcompletion/088d789· pfgen-bench (Preferred Networks) data docs0.586
74Qwen1.5-72B-Chatqa/chat/d1092d0· pfgen-bench (Preferred Networks) data docs0.582
75Qwen3 4B Basecompletion/af29cd4· pfgen-bench (Preferred Networks) data docs0.577
76Qwen-14B-Chatcompletion/bad8c09· pfgen-bench (Preferred Networks) data docs0.572
77Qwen2.5 7Bcompletion/4d123a9· pfgen-bench (Preferred Networks) data docs0.563
78Qwen2.5 7B Instructqa/chat/81e4cf3· pfgen-bench (Preferred Networks) data docs0.557
79DeepSeek LLM 67B Chatqa/chat/e1b18f0· pfgen-bench (Preferred Networks) data docs0.555
80GLM 4 9B 0414completion/77a535b· pfgen-bench (Preferred Networks) data docs0.554
81Mistral Nemocompletion/e27a5da· pfgen-bench (Preferred Networks) data docs0.548
82Qwen2 7B Instructqa/chat/443a1af· pfgen-bench (Preferred Networks) data docs0.547
83Gemma 2 27B ITqa/chat/3329a88· pfgen-bench (Preferred Networks) data docs0.545
84Qwen1.5-32B-Chatcompletion/146f406· pfgen-bench (Preferred Networks) data docs0.538
85Gemma 2 2B ITcompletion/a5a3763· pfgen-bench (Preferred Networks) data docs0.536
86Qwen2.5 7B Instructqa/chat/20023d7· pfgen-bench (Preferred Networks) data docs0.535
87gemma-7bcompletion/ebc12f4· pfgen-bench (Preferred Networks) data docs0.533
88Qwen 2 (7B)completion/2fd6882· pfgen-bench (Preferred Networks) data docs0.532
89Qwen2 7B Instructcompletion/880bb2d· pfgen-bench (Preferred Networks) data docs0.531
90Llama 3 8B Instructcompletion/93af24a· pfgen-bench (Preferred Networks) data docs0.523
91Qwen-14B-Chatqa/chat/7fee9c7· pfgen-bench (Preferred Networks) data docs0.522
92Llama-2-70b-chat-hfcompletion/152cf31· pfgen-bench (Preferred Networks) data docs0.519
93mixtral-8x22b-v0.1completion/40ef626· pfgen-bench (Preferred Networks) data docs0.517
94Phi-3 Small 8K Instructcompletion/02b97c5· pfgen-bench (Preferred Networks) data docs0.514
95Yi 1.5 34B Chatqa/chat/868c125· pfgen-bench (Preferred Networks) data docs0.514
96Llama 3.1 8Bcompletion/1568d43· pfgen-bench (Preferred Networks) data docs0.512
97Gemma 1.1 7B ITcompletion/edc5e6e· pfgen-bench (Preferred Networks) data docs0.509
98Qwen1.5-14Bcompletion/4e006c4· pfgen-bench (Preferred Networks) data docs0.509
99Llama 3 8B Instructqa/chat/3d447ac· pfgen-bench (Preferred Networks) data docs0.508
100Meta Llama 3 8bcompletion/0f9faf5· pfgen-bench (Preferred Networks) data docs0.505

Showing 51–100 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)