Benchmark
WeirdML v3 official score
Agentic benchmarks measure how well models carry out tasks that take several steps. Higher scores are better.
- Models measured
- 27
- Scores recorded
- 29
- Sources
- 1
- IQ weight
- 2.33
Score distribution
- 0.153
- Mean
- 0.099
- Median
- 0.115
- Std dev
- 0.05
- 10th percentile
- 0.312
- 90th percentile
- 0.0375
- Min
This benchmark has a weight of 2.33 for chat models, measured over 27 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | |
|---|---|---|
| 1 | Claude Fable 5.2config: fable-5.2, agent: claude_code, harness: claude_code@2.1.321, effort: xhigh· WeirdML v3 data | 0.431 |
| 2 | GPT-6 Astraconfig: gpt-6-astra-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.154.0, effort: xhigh· WeirdML v3 data | 0.422 |
| 3 | GPT-6.1 Solconfig: gpt-6.1-sol-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data | 0.383 |
| 4 | Claude Opus 5.5config: opus-5.5, agent: claude_code, harness: claude_code@2.1.280, effort: xhigh· WeirdML v3 data | 0.312 |
| 5 | GPT-6.1 Terraconfig: gpt-6.1-terra-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.164.0, effort: xhigh· WeirdML v3 data | 0.281 |
| 6 | Claude Fable 5.1config: fable-5.1, agent: claude_code, harness: claude_code@2.1.270, effort: xhigh· WeirdML v3 data | 0.26 |
| 7 | Gemini 4 Argonconfig: gemini-4-argon-aistudio-high, agent: gemini_cli, harness: gemini_cli@0.62.0, effort: high· WeirdML v3 data | 0.203 |
| 8 | Claude Sonnet 5.5config: sonnet-5.5, agent: claude_code, harness: claude_code@2.1.280, effort: xhigh· WeirdML v3 data | 0.199 |
| 9 | GPT-6 Solconfig: gpt-6-sol-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data | 0.197 |
| 10 | Claude Opus 5config: opus-5, agent: claude_code, harness: claude_code@2.1.270, effort: xhigh· WeirdML v3 data | 0.19 |
| 11 | Claude Fable 5config: fable-5, agent: claude_code, harness: claude_code@2.1.270, effort: xhigh· WeirdML v3 data | 0.156 |
| 12 | Muse Wonderconfig: muse-wonder-meta-xhigh-codex, agent: codex_cli, harness: codex_cli@0.162.1, effort: xhigh· WeirdML v3 data | 0.151 |
| 13 | GPT-5.6 Solconfig: gpt-5.6-sol-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.154.0, effort: xhigh· WeirdML v3 data | 0.15 |
| 14 | Claude Haiku 5.5config: haiku-5.5, agent: claude_code, harness: claude_code@2.1.280, effort: xhigh· WeirdML v3 data | 0.117 |
| 15 | Kimi K3.1 Proconfig: kimi-k3.1-pro-fireworks-xhigh-kimi-code, agent: kimi_code, harness: kimi_code@2.1.1, effort: xhigh· WeirdML v3 data | 0.0989 |
| 16 | DeepSeek V4.1 Flash 1017config: deepseek-v4.1-flash-1017-inferencenet-high-codex, agent: codex_cli, harness: codex_cli@0.162.1, effort: high· WeirdML v3 data | 0.091 |
| 17 | Grok 4.7config: grok-4.7-xai-xhigh-codex, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data | 0.0845 |
| 18 | GPT-6 Lunaconfig: gpt-6-luna-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data | 0.0787 |
| 19 | Kimi K3config: kimi-k3-fireworks-high-kimi-code, agent: kimi_code, harness: kimi_code@2.0.0, effort: high· WeirdML v3 data | 0.0754 |
| 20 | Gemini 3.8 Flashconfig: gemini-3.8-flash-aistudio-high, agent: gemini_cli, harness: gemini_cli@0.59.0, effort: high· WeirdML v3 data | 0.0738 |
| 21 | Muse Spark 1.3config: muse-spark-1.3-meta-xhigh-codex, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data | 0.0665 |
| 22 | Claude Opus 4.5config: opus-4.5, agent: claude_code, harness: claude_code@2.1.270, effort: high· WeirdML v3 data | 0.0651 |
| 23 | DeepSeek V4.1 Flashconfig: deepseek-v4.1-flash-novita-high-opencode, agent: opencode, harness: opencode@1.18.30, effort: high· WeirdML v3 data | 0.0595 |
| 24 | DeepSeek V4.1 Flashconfig: deepseek-v4.1-flash-novita-high-codex, agent: codex_cli, harness: codex_cli@0.156.0, effort: high· WeirdML v3 data | 0.0583 |
| 25 | GPT-5.6 Lunaconfig: gpt-5.6-luna-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.154.0, effort: xhigh· WeirdML v3 data | 0.0516 |
| 26 | GPT 5.2config: gpt-5.2-xhigh-openrouter, agent: codex_cli, harness: codex_cli@0.156.0, effort: xhigh· WeirdML v3 data | 0.0501 |
| 27 | Mistral Large 4config: mistral-large-4-mistral-high-opencode, agent: opencode, harness: opencode@1.18.30, effort: high· WeirdML v3 data | 0.042 |
| 28 | Inkling FP4config: inkling-together-xhigh-opencode, agent: opencode, harness: opencode@1.18.30, effort: xhigh· WeirdML v3 data | 0.039 |
| 29 | Inkling FP4config: inkling-deepinfra-xhigh-opencode, agent: opencode, harness: opencode@1.19.0, effort: xhigh· WeirdML v3 data | 0.0375 |
Tokens the model generated per task, as measured by the source. Comparable within this benchmark only.
Data credits: WeirdML v3 by Håvard Tveit Ihle (Norwegian Defence Research Establishment). API costs supported by Epoch AI, METR and NDRE.