Skip to content

Benchmark

Claw-Eval v1.1 · Multi-turn (38) · Pass@3

Scores are percentages. Higher is better.

Models and systems measured
24
Scores recorded
24
Sources
1
IQ weight
1.96

Score distribution

79.1%
Mean
84.2%
Median
17.3%
Std dev
60.5%
10th percentile
92.1%
90th percentile
13.2%
Min
13.2%92.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.96 for chat models, measured over 19 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
1Gemini 3.1 Pro PreviewClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs92.1%
2MiMo V2 ProClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs92.1%
3MiMo V2.5Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs92.1%
4Claude Opus 4.6Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs89.5%
5Claude Sonnet 4.6Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs89.5%
6DeepSeek V4 Pro 0813Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs89.5%
7GPT 5.4Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs89.5%
8Kimi K2.6Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs89.5%
9MiMo V2.5 ProClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs86.8%
10Qwen3.6 PlusClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs86.8%
11Gemini 3 FlashClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs84.2%
12MiniMax M2.7Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs84.2%
13DeepSeek V4 FlashClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs81.6%
14GLM 5.1Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs81.6%
15Kimi K2.5Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs81.6%
16Qwen3.5 397B A17BClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs76.3%
17MiMo V2 OmniClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs63.2%
18DeepSeek V3.2Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs60.5%
19Agnes 2.0 FlashClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs50%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
GLM 5 TurboClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs84.2%
GLM 5V TurboClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs86.8%
Mach-Mind 4 ProClaw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs71.1%
Muse Spark (version unspecified)Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs81.6%
Nemotron 3 Super (variant unspecified)Claw-Eval v1.1 · multi-turn · 38 tasks · N=3 · grader + user-agent: Claude Opus 4.6· Reported by Claw-Eval· via BenchBoard docs13.2%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.