Skip to content

Benchmark

Shaberi v2.1 (GPT5.1 judge): Rakuda

People compare model answers and choose which they prefer. Higher scores are better.

Models measured
44
Scores recorded
47
Sources
1
IQ weight
0.91

Score distribution

5.4
Mean
5.3
Median
2.6
Std dev
1.9
10th percentile
9
90th percentile
1
Min
19.4
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.91 for chat models, measured over 44 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Qwen3.6 Plusfree· Shaberi v2.1 (Shisa.AI) data docs9.4
2Gemini 2.5 Pro· Shaberi v2.1 (Shisa.AI) data docs9.2
3Gemini 3 Pro Preview· Shaberi v2.1 (Shisa.AI) data docs9.2
4Gemini 2.5 Flash· Shaberi v2.1 (Shisa.AI) data docs9.1
5Gemini 3 Flash· Shaberi v2.1 (Shisa.AI) data docs9
6Qwen3.5 397B A17B· Shaberi v2.1 (Shisa.AI) data docs9
7Gemma 4 26B A4B· Shaberi v2.1 (Shisa.AI) data docs8.7
8Gemma 4 31B IT· Shaberi v2.1 (Shisa.AI) data docs8.6
9Qwen3 235B A22B Instruct 2507· Shaberi v2.1 (Shisa.AI) data docs8.5
10GPT 4o· Shaberi v2.1 (Shisa.AI) data docs8
11Kimi K2 Instruct· Shaberi v2.1 (Shisa.AI) data docs8
12Gemma 4 E4B-it· Shaberi v2.1 (Shisa.AI) data docs7.9
13DeepSeek V3.1 Terminus· Shaberi v2.1 (Shisa.AI) data docs7.8
14Kimi K2.5· Shaberi v2.1 (Shisa.AI) data docs7.6
15Qwen3.6 27Bnothink· Shaberi v2.1 (Shisa.AI) data docs7.5
16GPT OSS 120B· Shaberi v2.1 (Shisa.AI) data docs7.4
17Qwen3 30B A3B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.9
18Llama 3.3 70B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.1
19Llama 3.1 405B Instruct· Shaberi v2.1 (Shisa.AI) data docs5.9
20Ling 1t· Shaberi v2.1 (Shisa.AI) data docs5.9
21Qwen3.6 27B32k· Shaberi v2.1 (Shisa.AI) data docs5.9
22GLM 4.7 Flash· Shaberi v2.1 (Shisa.AI) data docs5.5
23Gemma 3 27B IT· Shaberi v2.1 (Shisa.AI) data docs5.4
24Ministral 3 14B Instruct· Shaberi v2.1 (Shisa.AI) data docs5.3
25Nemotron 3 Nano 30B A3B· Shaberi v2.1 (Shisa.AI) data docs5.1
26Ministral 3 8b Instruct· Shaberi v2.1 (Shisa.AI) data docs4.9
27Llama 4 Scout· Shaberi v2.1 (Shisa.AI) data docs4.9
28Qwen3 8B· Shaberi v2.1 (Shisa.AI) data docs4.8
29Phi-4-reasoning· Shaberi v2.1 (Shisa.AI) data docs4.5
30GPT OSS 20b· Shaberi v2.1 (Shisa.AI) data docs4.4
31Phi-4-reasoningnothink· Shaberi v2.1 (Shisa.AI) data docs4.2
32Qwen3.5 122B-A10B· Shaberi v2.1 (Shisa.AI) data docs4.2
33Ling Flash 2.0· Shaberi v2.1 (Shisa.AI) data docs4
34ERNIE-4.5-21B-A3B-PT· Shaberi v2.1 (Shisa.AI) data docs3.9
35Phi 4 Multimodal Instruct· Shaberi v2.1 (Shisa.AI) data docs3.8
36Llama 3.1 8B Instruct· Shaberi v2.1 (Shisa.AI) data docs3.7
37Ministral 3 3b Instruct· Shaberi v2.1 (Shisa.AI) data docs3.4
38Phi 4 Mini· Shaberi v2.1 (Shisa.AI) data docs3.3
39ERNIE-4.5-VL-28B-A3B-PT· Shaberi v2.1 (Shisa.AI) data docs2.9
40Llama 3.2 3B· Shaberi v2.1 (Shisa.AI) data docs2.5
41Llama 3.2 1B Instruct· Shaberi v2.1 (Shisa.AI) data docs1.9
42Olmo 3 7B Instruct· Shaberi v2.1 (Shisa.AI) data docs1.9
43Mistral (7B) Instruct v0.1· Shaberi v2.1 (Shisa.AI) data docs1.6
44Nemotron 3 Nano 30B A3Bnothink· Shaberi v2.1 (Shisa.AI) data docs1
45Qwen3.5 27B· Shaberi v2.1 (Shisa.AI) data docs1
46Qwen3.5 35B A3B· Shaberi v2.1 (Shisa.AI) data docs1
47Qwen3.5 9B· Shaberi v2.1 (Shisa.AI) data docs1

Data credits: Shaberi v2.1, Shisa.AI, Apache-2.0. GPT-5.1-judge scores only; judges are not comparable. (licence)