Skip to content

Benchmark

WeirdML v3 official score

Agentic benchmarks measure how well models carry out tasks that take several steps. Higher scores are better.

Models measured
27
Scores recorded
29
Sources
1
IQ weight
2.33

Score distribution

0.153
Mean
0.099
Median
0.115
Std dev
0.05
10th percentile
0.312
90th percentile
0.0375
Min
0.03750.431
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.33 for chat models, measured over 27 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model
1Claude Fable 5.2config: fable-5.2, agent: claude_code, harness: claude_code@2.1.321, effort: xhigh· WeirdML v3 data0.431
2GPT-6 Astraconfig: gpt-6-astra-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.154.0, effort: xhigh· WeirdML v3 data0.422
3GPT-6.1 Solconfig: gpt-6.1-sol-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data0.383
4Claude Opus 5.5config: opus-5.5, agent: claude_code, harness: claude_code@2.1.280, effort: xhigh· WeirdML v3 data0.312
5GPT-6.1 Terraconfig: gpt-6.1-terra-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.164.0, effort: xhigh· WeirdML v3 data0.281
6Claude Fable 5.1config: fable-5.1, agent: claude_code, harness: claude_code@2.1.270, effort: xhigh· WeirdML v3 data0.26
7Gemini 4 Argonconfig: gemini-4-argon-aistudio-high, agent: gemini_cli, harness: gemini_cli@0.62.0, effort: high· WeirdML v3 data0.203
8Claude Sonnet 5.5config: sonnet-5.5, agent: claude_code, harness: claude_code@2.1.280, effort: xhigh· WeirdML v3 data0.199
9GPT-6 Solconfig: gpt-6-sol-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data0.197
10Claude Opus 5config: opus-5, agent: claude_code, harness: claude_code@2.1.270, effort: xhigh· WeirdML v3 data0.19
11Claude Fable 5config: fable-5, agent: claude_code, harness: claude_code@2.1.270, effort: xhigh· WeirdML v3 data0.156
12Muse Wonderconfig: muse-wonder-meta-xhigh-codex, agent: codex_cli, harness: codex_cli@0.162.1, effort: xhigh· WeirdML v3 data0.151
13GPT-5.6 Solconfig: gpt-5.6-sol-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.154.0, effort: xhigh· WeirdML v3 data0.15
14Claude Haiku 5.5config: haiku-5.5, agent: claude_code, harness: claude_code@2.1.280, effort: xhigh· WeirdML v3 data0.117
15Kimi K3.1 Proconfig: kimi-k3.1-pro-fireworks-xhigh-kimi-code, agent: kimi_code, harness: kimi_code@2.1.1, effort: xhigh· WeirdML v3 data0.0989
16DeepSeek V4.1 Flash 1017config: deepseek-v4.1-flash-1017-inferencenet-high-codex, agent: codex_cli, harness: codex_cli@0.162.1, effort: high· WeirdML v3 data0.091
17Grok 4.7config: grok-4.7-xai-xhigh-codex, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data0.0845
18GPT-6 Lunaconfig: gpt-6-luna-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data0.0787
19Kimi K3config: kimi-k3-fireworks-high-kimi-code, agent: kimi_code, harness: kimi_code@2.0.0, effort: high· WeirdML v3 data0.0754
20Gemini 3.8 Flashconfig: gemini-3.8-flash-aistudio-high, agent: gemini_cli, harness: gemini_cli@0.59.0, effort: high· WeirdML v3 data0.0738
21Muse Spark 1.3config: muse-spark-1.3-meta-xhigh-codex, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data0.0665
22Claude Opus 4.5config: opus-4.5, agent: claude_code, harness: claude_code@2.1.270, effort: high· WeirdML v3 data0.0651
23DeepSeek V4.1 Flashconfig: deepseek-v4.1-flash-novita-high-opencode, agent: opencode, harness: opencode@1.18.30, effort: high· WeirdML v3 data0.0595
24DeepSeek V4.1 Flashconfig: deepseek-v4.1-flash-novita-high-codex, agent: codex_cli, harness: codex_cli@0.156.0, effort: high· WeirdML v3 data0.0583
25GPT-5.6 Lunaconfig: gpt-5.6-luna-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.154.0, effort: xhigh· WeirdML v3 data0.0516
26GPT 5.2config: gpt-5.2-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data0.0501
27Mistral Large 4config: mistral-large-4-mistral-high-opencode, agent: opencode, harness: opencode@1.18.30, effort: high· WeirdML v3 data0.042
28Inkling FP4config: inkling-together-xhigh-opencode, agent: opencode, harness: opencode@1.18.30, effort: xhigh· WeirdML v3 data0.039
29Inkling FP4config: inkling-deepinfra-xhigh-opencode, agent: opencode, harness: opencode@1.19.0, effort: xhigh· WeirdML v3 data0.0375

Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.

Data credits: WeirdML v3 by Håvard Tveit Ihle (Norwegian Defence Research Establishment). API costs supported by Epoch AI, METR and NDRE.