Skip to content

Benchmark

Shaberi v2.1 (GPT5.1 judge): Tengu

People compare model answers and choose which they prefer. Higher scores are better.

Models measured
44
Scores recorded
47
Sources
1
IQ weight
1.15

Score distribution

5.5
Mean
5.9
Median
2.4
Std dev
2.2
10th percentile
8.1
90th percentile
0.139
Min
0.1399.1
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.15 for chat models, measured over 44 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 3 Pro Preview· Shaberi v2.1 (Shisa.AI) data docs9.1
2Gemini 3 Flash· Shaberi v2.1 (Shisa.AI) data docs8.9
3Gemini 2.5 Pro· Shaberi v2.1 (Shisa.AI) data docs8.9
4Qwen3.6 Plusfree· Shaberi v2.1 (Shisa.AI) data docs8.8
5Gemini 2.5 Flash· Shaberi v2.1 (Shisa.AI) data docs8.5
6Gemma 4 31B IT· Shaberi v2.1 (Shisa.AI) data docs8.1
7DeepSeek V3.1 Terminus· Shaberi v2.1 (Shisa.AI) data docs7.7
8Qwen3 235B A22B Instruct 2507· Shaberi v2.1 (Shisa.AI) data docs7.7
9Qwen3.5 397B A17B· Shaberi v2.1 (Shisa.AI) data docs7.6
10Gemma 4 26B A4B· Shaberi v2.1 (Shisa.AI) data docs7.6
11GPT 4o· Shaberi v2.1 (Shisa.AI) data docs7.5
12GPT OSS 120B· Shaberi v2.1 (Shisa.AI) data docs7.3
13Kimi K2 Instruct· Shaberi v2.1 (Shisa.AI) data docs7.3
14Kimi K2.5· Shaberi v2.1 (Shisa.AI) data docs7.3
15Qwen3.6 27Bnothink· Shaberi v2.1 (Shisa.AI) data docs7
16Gemma 4 E4B-it· Shaberi v2.1 (Shisa.AI) data docs6.8
17Gemma 3 27B IT· Shaberi v2.1 (Shisa.AI) data docs6.6
18Qwen3 30B A3B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.6
19Ling 1t· Shaberi v2.1 (Shisa.AI) data docs6.6
20Llama 3.3 70B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.5
21Ministral 3 14B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.5
22Llama 3.1 405B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.4
23Ministral 3 8b Instruct· Shaberi v2.1 (Shisa.AI) data docs6.3
24Ling Flash 2.0· Shaberi v2.1 (Shisa.AI) data docs5.9
25Nemotron 3 Nano 30B A3B· Shaberi v2.1 (Shisa.AI) data docs5.8
26Qwen3 8B· Shaberi v2.1 (Shisa.AI) data docs5.5
27ERNIE-4.5-21B-A3B-PT· Shaberi v2.1 (Shisa.AI) data docs5.4
28GPT OSS 20b· Shaberi v2.1 (Shisa.AI) data docs5.3
29GLM 4.7 Flash· Shaberi v2.1 (Shisa.AI) data docs5.3
30Ministral 3 3b Instruct· Shaberi v2.1 (Shisa.AI) data docs5.3
31Llama 3.1 8B Instruct· Shaberi v2.1 (Shisa.AI) data docs5
32Phi-4-reasoning· Shaberi v2.1 (Shisa.AI) data docs4.5
33Phi 4 Multimodal Instruct· Shaberi v2.1 (Shisa.AI) data docs4.4
34Phi 4 Mini· Shaberi v2.1 (Shisa.AI) data docs4.4
35ERNIE-4.5-VL-28B-A3B-PT· Shaberi v2.1 (Shisa.AI) data docs4.2
36Phi-4-reasoningnothink· Shaberi v2.1 (Shisa.AI) data docs4.1
37Llama 4 Scout· Shaberi v2.1 (Shisa.AI) data docs4
38Qwen3.5 122B-A10B· Shaberi v2.1 (Shisa.AI) data docs3.7
39Llama 3.2 3B· Shaberi v2.1 (Shisa.AI) data docs3.4
40Olmo 3 7B Instruct· Shaberi v2.1 (Shisa.AI) data docs3.3
41Qwen3.6 27B32k· Shaberi v2.1 (Shisa.AI) data docs2.8
42Llama 3.2 1B Instruct· Shaberi v2.1 (Shisa.AI) data docs2.2
43Mistral (7B) Instruct v0.1· Shaberi v2.1 (Shisa.AI) data docs2.1
44Qwen3.5 35B A3B· Shaberi v2.1 (Shisa.AI) data docs1.3
45Qwen3.5 27B· Shaberi v2.1 (Shisa.AI) data docs0.191
46Qwen3.5 9B· Shaberi v2.1 (Shisa.AI) data docs0.157
47Nemotron 3 Nano 30B A3Bnothink· Shaberi v2.1 (Shisa.AI) data docs0.139

Data credits: Shaberi v2.1, Shisa.AI, Apache-2.0. GPT-5.1-judge scores only; judges are not comparable. (licence)