Skip to content

Benchmark

Roomote Role Evals

Scores are percentages.

Models measured
21
Scores recorded
115
Sources
1
IQ weight
1.68

Score distribution

65.7%
Mean
73%
Median
28.2%
Std dev
27%
10th percentile
96%
90th percentile
0%
Min
0%100%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.68 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
51DeepSeek V4 Procoding· Roomote Role Evals data79%
52MiniMax M3vision-extraction· Roomote Role Evals data79%
53GPT-5.6 Terraadvisor· Roomote Role Evals data78%
54Claude Sonnet 5explorer· Roomote Role Evals data77%
55Claude Fable 5advisor· Roomote Role Evals data75%
56GPT-5.6 Lunaadvisor· Roomote Role Evals data75%
57Gemini 3.6 Flashcoding· Roomote Role Evals data74%
58Claude Fable 5explorer· Roomote Role Evals data73%
59Kimi K2.7 Coderouter-lookup· Roomote Role Evals data73%
60Inkling FP4explorer· Roomote Role Evals data72%
61Gemini 3.7 Flashadvisor· Roomote Role Evals data71%
62Grok 4.5explorer· Roomote Role Evals data71%
63GLM 5.2explorer· Roomote Role Evals data70%
64DeepSeek V4 Flash 0731explorer· Roomote Role Evals data69%
65DeepSeek V4 Pro 0813explorer· Roomote Role Evals data69%
66Grok 4.6advisor· Roomote Role Evals data67%
67Gemini 3.6 Flashadvisor· Roomote Role Evals data66%
68Claude Sonnet 5advisor· Roomote Role Evals data65%
69Kimi K3advisor· Roomote Role Evals data65%
70Kimi K2.7 Codevision-extraction· Roomote Role Evals data64%
71Qwen3.8 Maxvision-extraction· Roomote Role Evals data64%
72Claude Opus 5explorer· Roomote Role Evals data62%
73DeepSeek V4 Proadvisor· Roomote Role Evals data62%
74Kimi K2.7 Codeadvisor· Roomote Role Evals data61%
75Claude Haiku 4.5explorer· Roomote Role Evals data60%
76Claude Fable 5review· Roomote Role Evals data57%
77Kimi K3explorer· Roomote Role Evals data55%
78Kimi K3review· Roomote Role Evals data54%
79Kimi K2.7 Codeexplorer· Roomote Role Evals data53%
80Qwen3.8 Maxexplorer· Roomote Role Evals data53%
81DeepSeek V4 Proexplorer· Roomote Role Evals data52%
82Claude Haiku 4.5advisor· Roomote Role Evals data51%
83Claude Opus 5review· Roomote Role Evals data51%
84GLM 5.2review· Roomote Role Evals data51%
85GPT-5.6 Terrareview· Roomote Role Evals data50%
86MiniMax M3advisor· Roomote Role Evals data49%
87MiniMax M3explorer· Roomote Role Evals data47%
88GLM 5.2advisor· Roomote Role Evals data45%
89Grok 4.5review· Roomote Role Evals data42%
90Gemini 3.7 Flashreview· Roomote Role Evals data40%
91GPT-5.6 Lunareview· Roomote Role Evals data40%
92Claude Sonnet 5review· Roomote Role Evals data39%
93Qwen3.8 Maxadvisor· Roomote Role Evals data39%
94DeepSeek V4 Flash 0731review· Roomote Role Evals data38%
95Claude Haiku 4.5coding· Roomote Role Evals data36%
96DeepSeek V4 Flash 0731advisor· Roomote Role Evals data35%
97Qwen3.8 Maxreview· Roomote Role Evals data35%
98Claude Haiku 4.5review· Roomote Role Evals data34%
99DeepSeek V4 Pro 0813advisor· Roomote Role Evals data34%
100GPT-5.6 Solreview· Roomote Role Evals data34%

Showing 51–100 of 115 results.