Skip to content

Benchmark

pfgen-bench Japanese generation: Score

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
1.59

Score distribution

0.498
Mean
0.483
Median
0.202
Std dev
0.256
10th percentile
0.804
90th percentile
0.0006
Min
0.00060.958
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.59 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51Qwen2.5 32Bcompletion/b356bba· pfgen-bench (Preferred Networks) data docs0.587
52Phi-3 Medium 4K Instructcompletion/74cf0cd· pfgen-bench (Preferred Networks) data docs0.578
53Phi-3 Medium 128K Instructcompletion/3c9dc10· pfgen-bench (Preferred Networks) data docs0.574
54Gemma 2 9B ITcompletion/d29837e· pfgen-bench (Preferred Networks) data docs0.574
55Mistral Small 24B Instructcompletion/0b0698f· pfgen-bench (Preferred Networks) data docs0.571
56Llama 3.1 70Bcompletion/19cc458· pfgen-bench (Preferred Networks) data docs0.566
57Llama 3 70B Instructqa/chat/3c91259· pfgen-bench (Preferred Networks) data docs0.566
58Phi-3 Medium 4K Instructqa/chat/d396e2e· pfgen-bench (Preferred Networks) data docs0.565
59Meta Llama 3 70bcompletion/a1448c0· pfgen-bench (Preferred Networks) data docs0.562
60Qwen3 8B Basecompletion/57fa665· pfgen-bench (Preferred Networks) data docs0.56
61Mistral Small 24B Instructqa/chat/f9649b1· pfgen-bench (Preferred Networks) data docs0.559
62Llama 3 70B Instructcompletion/8dfb479· pfgen-bench (Preferred Networks) data docs0.557
63Mistral Small 24B Instructqa/chat/ecbf52c· pfgen-bench (Preferred Networks) data docs0.554
64Gemma 2 27B ITcompletion/a3fa605· pfgen-bench (Preferred Networks) data docs0.551
65Llama 3.1 70B Instructcompletion/12b7638· pfgen-bench (Preferred Networks) data docs0.543
66Qwen2.5 14Bcompletion/174d4f0· pfgen-bench (Preferred Networks) data docs0.542
67Phi-3 Medium 128K Instructqa/chat/016fa5e· pfgen-bench (Preferred Networks) data docs0.539
68Llama 3.1 70B Instructqa/chat/011aba5· pfgen-bench (Preferred Networks) data docs0.538
69Gemma 2 27B ITqa/chat/3329a88· pfgen-bench (Preferred Networks) data docs0.535
70Qwen2.5 7B Instructqa/chat/81e4cf3· pfgen-bench (Preferred Networks) data docs0.512
71Qwen2 57B A14B Instructqa/chat/d1e4d61· pfgen-bench (Preferred Networks) data docs0.507
72gemma-2-9bcompletion/161d30c· pfgen-bench (Preferred Networks) data docs0.501
73Qwen1.5-72B-Chatqa/chat/d1092d0· pfgen-bench (Preferred Networks) data docs0.5
74Qwen2.5 7B Instructqa/chat/20023d7· pfgen-bench (Preferred Networks) data docs0.497
75Qwen1.5-72B-Chatcompletion/088d789· pfgen-bench (Preferred Networks) data docs0.496
76Gemma 2 9B ITqa/chat/37ba4f6· pfgen-bench (Preferred Networks) data docs0.487
77Qwen2 57B A14B Instructcompletion/bafc717· pfgen-bench (Preferred Networks) data docs0.486
78Qwen1.5-72Bcompletion/a4094f9· pfgen-bench (Preferred Networks) data docs0.483
79Qwen3 4B Basecompletion/af29cd4· pfgen-bench (Preferred Networks) data docs0.478
80Qwen2.5 7Bcompletion/4d123a9· pfgen-bench (Preferred Networks) data docs0.467
81DeepSeek LLM 67B Chatqa/chat/e1b18f0· pfgen-bench (Preferred Networks) data docs0.466
82Qwen-14B-Chatcompletion/bad8c09· pfgen-bench (Preferred Networks) data docs0.462
83Gemma 2 2B ITcompletion/a5a3763· pfgen-bench (Preferred Networks) data docs0.459
84GLM 4 9B 0414completion/77a535b· pfgen-bench (Preferred Networks) data docs0.454
85Qwen2 7B Instructqa/chat/443a1af· pfgen-bench (Preferred Networks) data docs0.449
86Gemma 1.1 7B ITcompletion/edc5e6e· pfgen-bench (Preferred Networks) data docs0.438
87Qwen1.5-32B-Chatcompletion/146f406· pfgen-bench (Preferred Networks) data docs0.437
88Qwen2 7B Instructcompletion/880bb2d· pfgen-bench (Preferred Networks) data docs0.433
89Gemma 2 2B ITqa/chat/3d0cfea· pfgen-bench (Preferred Networks) data docs0.433
90Qwen 2 (7B)completion/2fd6882· pfgen-bench (Preferred Networks) data docs0.432
91Qwen-14B-Chatqa/chat/7fee9c7· pfgen-bench (Preferred Networks) data docs0.43
92Mistral Nemocompletion/e27a5da· pfgen-bench (Preferred Networks) data docs0.427
93Gemma 1.1 7B ITqa/chat/7cedd4e· pfgen-bench (Preferred Networks) data docs0.427
94gemma-7bcompletion/ebc12f4· pfgen-bench (Preferred Networks) data docs0.416
95Qwen1.5-32B-Chatqa/chat/9df40bc· pfgen-bench (Preferred Networks) data docs0.415
96Yi 1.5 34B Chatqa/chat/868c125· pfgen-bench (Preferred Networks) data docs0.414
97Llama 3 8B Instructcompletion/93af24a· pfgen-bench (Preferred Networks) data docs0.414
98Llama 3.1 8Bcompletion/1568d43· pfgen-bench (Preferred Networks) data docs0.41
99Llama-2-70b-chat-hfcompletion/152cf31· pfgen-bench (Preferred Networks) data docs0.409
100Phi-3 Small 8K Instructcompletion/02b97c5· pfgen-bench (Preferred Networks) data docs0.409

Showing 51–100 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)