Skip to content

Benchmark

pfgen-bench Japanese generation: Fluency

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
1.56

Score distribution

0.573
Mean
0.557
Median
0.195
Std dev
0.351
10th percentile
0.839
90th percentile
0.001
Min
0.0011
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.56 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1DeepSeek V3completion/4a38349· pfgen-bench (Preferred Networks) data docs1
2Claude Opus 4qa/chat/2dafa10· pfgen-bench (Preferred Networks) data docs0.997
3Claude Opus 4qa/chat/d33465d· pfgen-bench (Preferred Networks) data docs0.974
4ChatGPT-4o Latestqa/chat/c275a7b· pfgen-bench (Preferred Networks) data docs0.954
5Claude 3.5 Sonnetqa/chat/2938dda· pfgen-bench (Preferred Networks) data docs0.949
6GPT 4oqa/chat/5e1e9f4· pfgen-bench (Preferred Networks) data docs0.919
7Claude 3 Opusqa/chat/89cdaa9· pfgen-bench (Preferred Networks) data docs0.911
8Qwen 3 235B A22Bcompletion/89946a4· pfgen-bench (Preferred Networks) data docs0.902
9GPT 4qa/chat/8cad1b9· pfgen-bench (Preferred Networks) data docs0.888
10Qwen2.5 72B Instructqa/chat/5e498db· pfgen-bench (Preferred Networks) data docs0.871
11Qwen Maxqa/chat/fa9558d· pfgen-bench (Preferred Networks) data docs0.864
12Qwen Maxqa/chat/0944f44· pfgen-bench (Preferred Networks) data docs0.862
13MiniMax Text 01qa/chat/74d4af0· pfgen-bench (Preferred Networks) data docs0.858
14MiniMax Text 01qa/chat/05c1dce· pfgen-bench (Preferred Networks) data docs0.85
15Qwen2.5 72B Instructqa/chat/5705734· pfgen-bench (Preferred Networks) data docs0.847
16DeepSeek R1qa/chat/5abaa36· pfgen-bench (Preferred Networks) data docs0.839
17Mistral Large Instruct 2411qa/chat/c6a3714· pfgen-bench (Preferred Networks) data docs0.828
18gemini-1.5-pro-002qa/chat/873c5b2· pfgen-bench (Preferred Networks) data docs0.826
19Mistral Large Instruct 2411qa/chat/564af5f· pfgen-bench (Preferred Networks) data docs0.822
20GPT-4 Turboqa/chat/e96542c· pfgen-bench (Preferred Networks) data docs0.82
21ChatGPT-4o Latestqa/chat/f029db6· pfgen-bench (Preferred Networks) data docs0.819
22Nemotron-4 340B Instructcompletion/e99a1d3· pfgen-bench (Preferred Networks) data docs0.816
23Claude 3 Sonnetqa/chat/f70c14d· pfgen-bench (Preferred Networks) data docs0.81
24Mistral Large Instruct 2411completion/b2f50ab· pfgen-bench (Preferred Networks) data docs0.81
25Nemotron-4 340B Instructqa/chat/0d4b478· pfgen-bench (Preferred Networks) data docs0.768
26Llama 3.1 405Bcompletion/d0c2319· pfgen-bench (Preferred Networks) data docs0.767
27Phi 4completion/118d26c· pfgen-bench (Preferred Networks) data docs0.754
28ChatGPT-4o Latestqa/chat/84cddfa· pfgen-bench (Preferred Networks) data docs0.751
29Qwen2.5 72Bcompletion/8c7477d· pfgen-bench (Preferred Networks) data docs0.749
30Claude 3 Haikuqa/chat/9e7a056· pfgen-bench (Preferred Networks) data docs0.747
31gemini-1.5-flash-002qa/chat/b868ea0· pfgen-bench (Preferred Networks) data docs0.742
32Qwen3 32Bcompletion/9f132a0· pfgen-bench (Preferred Networks) data docs0.736
33QwQ 32B Previewqa/chat/a362164· pfgen-bench (Preferred Networks) data docs0.73
34Gemini 2.0 Flash Experimentalqa/chat/5f7c6a9· pfgen-bench (Preferred Networks) data docs0.727
35QwQ 32B Previewqa/chat/a240792· pfgen-bench (Preferred Networks) data docs0.719
36Qwen3 30B A3b Basecompletion/8d6c0f4· pfgen-bench (Preferred Networks) data docs0.707
37Phi 4qa/chat/af856d5· pfgen-bench (Preferred Networks) data docs0.706
38Phi 4qa/chat/772e05d· pfgen-bench (Preferred Networks) data docs0.705
39Qwen 2 (72B)completion/ff6791f· pfgen-bench (Preferred Networks) data docs0.703
40Qwen2 72B Instructqa/chat/baf6c82· pfgen-bench (Preferred Networks) data docs0.703
41Qwen3 8Bcompletion/9d7e974· pfgen-bench (Preferred Networks) data docs0.695
42Qwen2.5 32Bcompletion/b356bba· pfgen-bench (Preferred Networks) data docs0.69
43Mistral Small 24B Instructcompletion/0b0698f· pfgen-bench (Preferred Networks) data docs0.684
44Qwen2 72B Instructcompletion/c1ddc4d· pfgen-bench (Preferred Networks) data docs0.684
45Qwen3 14B Basecompletion/3e8d9d4· pfgen-bench (Preferred Networks) data docs0.681
46Phi-3 Medium 4K Instructcompletion/74cf0cd· pfgen-bench (Preferred Networks) data docs0.675
47Gemma 2 9B ITcompletion/d29837e· pfgen-bench (Preferred Networks) data docs0.674
48Phi-3 Medium 128K Instructcompletion/3c9dc10· pfgen-bench (Preferred Networks) data docs0.674
49Llama 3 70B Instructqa/chat/3c91259· pfgen-bench (Preferred Networks) data docs0.665
50Llama 3.1 70Bcompletion/19cc458· pfgen-bench (Preferred Networks) data docs0.665

Showing 1–50 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)