Skip to content

Benchmark

Roomote Role Evals

Scores are percentages.

Models measured
21
Scores recorded
115
Sources
1
IQ weight
1.68

Score distribution

65.7%
Mean
73%
Median
28.2%
Std dev
27%
10th percentile
96%
90th percentile
0%
Min
0%100%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.68 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Gemini 3.6 Flashvision-extraction· Roomote Role Evals data100%
2Gemini 3.7 Flashrouter-lookup· Roomote Role Evals data100%
3GPT-5.6 Solrouter-lookup· Roomote Role Evals data100%
4Claude Fable 5router-lookup· Roomote Role Evals data98%
5Claude Opus 5router-lookup· Roomote Role Evals data98%
6Kimi K3router-lookup· Roomote Role Evals data98%
7Gemini 3.6 Flashrouter-lookup· Roomote Role Evals data97%
8GPT-5.6 Lunarouter-lookup· Roomote Role Evals data97%
9Claude Fable 5coding· Roomote Role Evals data96%
10Claude Opus 5coding· Roomote Role Evals data96%
11Claude Sonnet 5coding· Roomote Role Evals data96%
12GPT-5.6 Lunacoding· Roomote Role Evals data96%
13Grok 4.5coding· Roomote Role Evals data96%
14Kimi K3coding· Roomote Role Evals data96%
15Claude Sonnet 5router-lookup· Roomote Role Evals data95%
16GLM 5.2router-lookup· Roomote Role Evals data95%
17Claude Fable 5vision-extraction· Roomote Role Evals data93%
18Gemini 3.7 Flashvision-extraction· Roomote Role Evals data93%
19GPT-5.6 Solvision-extraction· Roomote Role Evals data93%
20GPT-5.6 Terrarouter-lookup· Roomote Role Evals data93%
21GPT-5.6 Terravision-extraction· Roomote Role Evals data93%
22Grok 4.5router-lookup· Roomote Role Evals data93%
23Kimi K3vision-extraction· Roomote Role Evals data93%
24Claude Haiku 4.5router-lookup· Roomote Role Evals data90%
25GLM 5.2coding· Roomote Role Evals data90%
26GPT-5.6 Solcoding· Roomote Role Evals data90%
27GPT-5.6 Solexplorer· Roomote Role Evals data90%
28GPT-5.6 Terracoding· Roomote Role Evals data90%
29Gemini 3.7 Flashexplorer· Roomote Role Evals data89%
30Qwen3.8 Maxcoding· Roomote Role Evals data89%
31DeepSeek V4 Pro 0813router-lookup· Roomote Role Evals data88%
32MiniMax M3router-lookup· Roomote Role Evals data87%
33Claude Haiku 4.5vision-extraction· Roomote Role Evals data86%
34Claude Opus 5vision-extraction· Roomote Role Evals data86%
35Claude Sonnet 5vision-extraction· Roomote Role Evals data86%
36GPT-5.6 Lunavision-extraction· Roomote Role Evals data86%
37Grok 4.5vision-extraction· Roomote Role Evals data86%
38MiniMax M3coding· Roomote Role Evals data86%
39DeepSeek V4 Prorouter-lookup· Roomote Role Evals data85%
40Gemini 3.6 Flashexplorer· Roomote Role Evals data85%
41Kimi K2.7 Codecoding· Roomote Role Evals data85%
42Qwen3.8 Maxrouter-lookup· Roomote Role Evals data85%
43GPT-5.6 Terraexplorer· Roomote Role Evals data84%
44Claude Opus 5advisor· Roomote Role Evals data83%
45DeepSeek V4 Flash 0731router-lookup· Roomote Role Evals data83%
46GPT-5.6 Lunaexplorer· Roomote Role Evals data83%
47DeepSeek V4 Flash 0731coding· Roomote Role Evals data81%
48Grok 4.6explorer· Roomote Role Evals data81%
49GPT-5.6 Soladvisor· Roomote Role Evals data80%
50Grok 4.5advisor· Roomote Role Evals data80%

Showing 1–50 of 115 results.