Skip to content

Benchmark

Shaberi v2.1 (GPT5.1 judge): JA-MT

People compare model answers and choose which they prefer. Higher scores are better.

Models measured
44
Scores recorded
47
Sources
1
IQ weight
0.96

Score distribution

5.7
Mean
6.1
Median
2.4
Std dev
2.2
10th percentile
8.6
90th percentile
1
Min
18.8
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.96 for chat models, measured over 44 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Qwen3.6 Plusfree· Shaberi v2.1 (Shisa.AI) data docs8.8
2Gemini 3 Flash· Shaberi v2.1 (Shisa.AI) data docs8.8
3Gemini 3 Pro Preview· Shaberi v2.1 (Shisa.AI) data docs8.8
4Qwen3.5 397B A17B· Shaberi v2.1 (Shisa.AI) data docs8.7
5Gemini 2.5 Pro· Shaberi v2.1 (Shisa.AI) data docs8.7
6Gemini 2.5 Flash· Shaberi v2.1 (Shisa.AI) data docs8.6
7Gemma 4 31B IT· Shaberi v2.1 (Shisa.AI) data docs8.3
8Gemma 4 26B A4B· Shaberi v2.1 (Shisa.AI) data docs8.1
9Qwen3 235B A22B Instruct 2507· Shaberi v2.1 (Shisa.AI) data docs8.1
10DeepSeek V3.1 Terminus· Shaberi v2.1 (Shisa.AI) data docs7.9
11GPT 4o· Shaberi v2.1 (Shisa.AI) data docs7.9
12GPT OSS 120B· Shaberi v2.1 (Shisa.AI) data docs7.9
13Qwen3 30B A3B Instruct· Shaberi v2.1 (Shisa.AI) data docs7.7
14Gemma 4 E4B-it· Shaberi v2.1 (Shisa.AI) data docs7.6
15Kimi K2.5· Shaberi v2.1 (Shisa.AI) data docs7.5
16Kimi K2 Instruct· Shaberi v2.1 (Shisa.AI) data docs7.5
17Ling 1t· Shaberi v2.1 (Shisa.AI) data docs7.1
18Qwen3.6 27Bnothink· Shaberi v2.1 (Shisa.AI) data docs6.9
19Gemma 3 27B IT· Shaberi v2.1 (Shisa.AI) data docs6.9
20Llama 3.3 70B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.6
21Llama 3.1 405B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.5
22Ministral 3 14B Instruct· Shaberi v2.1 (Shisa.AI) data docs6.4
23Ministral 3 8b Instruct· Shaberi v2.1 (Shisa.AI) data docs6.2
24Nemotron 3 Nano 30B A3B· Shaberi v2.1 (Shisa.AI) data docs6.1
25Qwen3 8B· Shaberi v2.1 (Shisa.AI) data docs6.1
26GPT OSS 20b· Shaberi v2.1 (Shisa.AI) data docs5.6
27Ling Flash 2.0· Shaberi v2.1 (Shisa.AI) data docs5
28ERNIE-4.5-21B-A3B-PT· Shaberi v2.1 (Shisa.AI) data docs4.9
29Ministral 3 3b Instruct· Shaberi v2.1 (Shisa.AI) data docs4.8
30Phi-4-reasoning· Shaberi v2.1 (Shisa.AI) data docs4.7
31Phi 4 Multimodal Instruct· Shaberi v2.1 (Shisa.AI) data docs4.7
32GLM 4.7 Flash· Shaberi v2.1 (Shisa.AI) data docs4.6
33Phi-4-reasoningnothink· Shaberi v2.1 (Shisa.AI) data docs4.4
34Llama 3.1 8B Instruct· Shaberi v2.1 (Shisa.AI) data docs4.3
35Llama 4 Scout· Shaberi v2.1 (Shisa.AI) data docs4.3
36Phi 4 Mini· Shaberi v2.1 (Shisa.AI) data docs4.1
37Qwen3.5 122B-A10B· Shaberi v2.1 (Shisa.AI) data docs3.8
38ERNIE-4.5-VL-28B-A3B-PT· Shaberi v2.1 (Shisa.AI) data docs3.5
39Llama 3.2 3B· Shaberi v2.1 (Shisa.AI) data docs3.3
40Qwen3.6 27B32k· Shaberi v2.1 (Shisa.AI) data docs3.1
41Olmo 3 7B Instruct· Shaberi v2.1 (Shisa.AI) data docs2.9
42Mistral (7B) Instruct v0.1· Shaberi v2.1 (Shisa.AI) data docs2.2
43Llama 3.2 1B Instruct· Shaberi v2.1 (Shisa.AI) data docs2
44Nemotron 3 Nano 30B A3Bnothink· Shaberi v2.1 (Shisa.AI) data docs1
45Qwen3.5 27B· Shaberi v2.1 (Shisa.AI) data docs1
46Qwen3.5 35B A3B· Shaberi v2.1 (Shisa.AI) data docs1
47Qwen3.5 9B· Shaberi v2.1 (Shisa.AI) data docs1

Data credits: Shaberi v2.1, Shisa.AI, Apache-2.0. GPT-5.1-judge scores only; judges are not comparable. (licence)