Skip to content

Benchmark

pfgen-bench Japanese generation: Score

Higher scores are better.

Models measured
101
Scores recorded
156
Sources
1
IQ weight
1.59

Score distribution

0.498
Mean
0.483
Median
0.202
Std dev
0.256
10th percentile
0.804
90th percentile
0.0006
Min
0.00060.958
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.59 for chat models, measured over 101 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Opus 4qa/chat/2dafa10· pfgen-bench (Preferred Networks) data docs0.958
2Claude Opus 4qa/chat/d33465d· pfgen-bench (Preferred Networks) data docs0.946
3DeepSeek V3completion/4a38349· pfgen-bench (Preferred Networks) data docs0.934
4ChatGPT-4o Latestqa/chat/c275a7b· pfgen-bench (Preferred Networks) data docs0.931
5Claude 3.5 Sonnetqa/chat/2938dda· pfgen-bench (Preferred Networks) data docs0.93
6GPT 4oqa/chat/5e1e9f4· pfgen-bench (Preferred Networks) data docs0.862
7DeepSeek R1qa/chat/5abaa36· pfgen-bench (Preferred Networks) data docs0.858
8ChatGPT-4o Latestqa/chat/f029db6· pfgen-bench (Preferred Networks) data docs0.841
9Qwen Maxqa/chat/fa9558d· pfgen-bench (Preferred Networks) data docs0.836
10Qwen Maxqa/chat/0944f44· pfgen-bench (Preferred Networks) data docs0.835
11MiniMax Text 01qa/chat/74d4af0· pfgen-bench (Preferred Networks) data docs0.828
12Claude 3 Opusqa/chat/89cdaa9· pfgen-bench (Preferred Networks) data docs0.827
13gemini-1.5-pro-002qa/chat/873c5b2· pfgen-bench (Preferred Networks) data docs0.819
14MiniMax Text 01qa/chat/05c1dce· pfgen-bench (Preferred Networks) data docs0.816
15Qwen 3 235B A22Bcompletion/89946a4· pfgen-bench (Preferred Networks) data docs0.813
16GPT-4 Turboqa/chat/e96542c· pfgen-bench (Preferred Networks) data docs0.804
17GPT 4qa/chat/8cad1b9· pfgen-bench (Preferred Networks) data docs0.792
18ChatGPT-4o Latestqa/chat/84cddfa· pfgen-bench (Preferred Networks) data docs0.784
19Qwen2.5 72B Instructqa/chat/5e498db· pfgen-bench (Preferred Networks) data docs0.783
20Qwen2.5 72B Instructqa/chat/5705734· pfgen-bench (Preferred Networks) data docs0.778
21gemini-1.5-flash-002qa/chat/b868ea0· pfgen-bench (Preferred Networks) data docs0.741
22Mistral Large Instruct 2411qa/chat/c6a3714· pfgen-bench (Preferred Networks) data docs0.733
23Claude 3 Sonnetqa/chat/f70c14d· pfgen-bench (Preferred Networks) data docs0.732
24Gemini 2.0 Flash Experimentalqa/chat/5f7c6a9· pfgen-bench (Preferred Networks) data docs0.727
25Mistral Large Instruct 2411qa/chat/564af5f· pfgen-bench (Preferred Networks) data docs0.726
26Nemotron-4 340B Instructcompletion/e99a1d3· pfgen-bench (Preferred Networks) data docs0.718
27Mistral Large Instruct 2411completion/b2f50ab· pfgen-bench (Preferred Networks) data docs0.709
28Nemotron-4 340B Instructqa/chat/0d4b478· pfgen-bench (Preferred Networks) data docs0.705
29Llama 3.1 405Bcompletion/d0c2319· pfgen-bench (Preferred Networks) data docs0.676
30Qwen3 32Bcompletion/9f132a0· pfgen-bench (Preferred Networks) data docs0.663
31Claude 3 Haikuqa/chat/9e7a056· pfgen-bench (Preferred Networks) data docs0.663
32Gemini 2.0 Flash Experimentalqa/chat/c246287, thinking· pfgen-bench (Preferred Networks) data docs0.659
33Qwen2.5 72Bcompletion/8c7477d· pfgen-bench (Preferred Networks) data docs0.649
34Qwen2 72B Instructqa/chat/baf6c82· pfgen-bench (Preferred Networks) data docs0.647
35Phi 4completion/118d26c· pfgen-bench (Preferred Networks) data docs0.642
36QwQ 32B Previewqa/chat/a362164· pfgen-bench (Preferred Networks) data docs0.641
37Qwen2.5 32B Instructqa/chat/01f44c1· pfgen-bench (Preferred Networks) data docs0.631
38Qwen2.5 32B Instructqa/chat/8a041a4· pfgen-bench (Preferred Networks) data docs0.63
39Phi 4qa/chat/af856d5· pfgen-bench (Preferred Networks) data docs0.63
40Phi 4qa/chat/772e05d· pfgen-bench (Preferred Networks) data docs0.629
41QwQ 32B Previewqa/chat/a240792· pfgen-bench (Preferred Networks) data docs0.629
42Qwen 2.5 14B Instructqa/chat/54fea07· pfgen-bench (Preferred Networks) data docs0.615
43GPT-3.5 Turboqa/chat/eaeade4· pfgen-bench (Preferred Networks) data docs0.614
44Qwen3 30B A3b Basecompletion/8d6c0f4· pfgen-bench (Preferred Networks) data docs0.611
45Qwen 2.5 14B Instructqa/chat/4a65a55· pfgen-bench (Preferred Networks) data docs0.606
46Qwen 2 (72B)completion/ff6791f· pfgen-bench (Preferred Networks) data docs0.606
47Qwen3 8Bcompletion/9d7e974· pfgen-bench (Preferred Networks) data docs0.605
48Qwen2 72B Instructcompletion/c1ddc4d· pfgen-bench (Preferred Networks) data docs0.603
49Qwen3 8Bqa/chat/667d843· pfgen-bench (Preferred Networks) data docs0.59
50Qwen3 14B Basecompletion/3e8d9d4· pfgen-bench (Preferred Networks) data docs0.587

Showing 1–50 of 156 results.

Data credits: pfgen-bench, Preferred Networks. Apache-2.0 except LLM-generated outputs; aggregate scores only. (licence)