Skip to content

Benchmark

AutomationBench

Scores are percentages. Higher is better.

Models and systems measured
41
Scores recorded
152
Sources
2
IQ weight
0.00

Score distribution

29.8%
Mean
30.3%
Median
13.5%
Std dev
13.5%
10th percentile
48.8%
90th percentile
1.2%
Min
1.2%61.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.00 for chat models, measured over 39 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
101GPT-5.6 SolAnthropic Fable/Mythos 5.1 launch table· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.19.6%
102GPT-5.6 Solmedium effort · $0.42 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.19.6%
103GPT-5.6 SolSystem Card capability summary · adaptive thinking · max effort · default sampling · 5 trials · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.19.6%
104GPT-5.6 SolOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.18.1%
105GPT-5.6 SolOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.18.1%
106GPT-5.6 SolOpus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.18.1%
107GPT-5.6 Sol· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim18.1%
108Claude Fable 5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim17.4%
109Claude Fable 5Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17.4%
110Claude Fable 5OpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17.4%
111Claude Fable 5OpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17.4%
112Claude Fable 5Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17.4%
113Claude Fable 5Anthropic Fable/Mythos 5.1 launch table· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17.1%
114Claude Fable 5System Card capability summary · adaptive thinking · max effort · default sampling · 5 trials · context ≤1M unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17.1%
115Claude Opus 4.8Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17%
116GPT-5.6 Lunamax effort · $0.07 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.17%
117Claude Opus 4.8Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.15.5%
118Claude Opus 4.8OpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.15.5%
119Claude Opus 4.8Opus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.15.5%
120GPT-5.6 Terra· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim15.2%
121GPT-5.6 TerraOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.15.2%
122GPT-5.6 Luna· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim14.9%
123GPT-5.6 LunaOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.14.9%
124Gemini 3.5 FlashFable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.14.5%
125Gemini 3.5 FlashOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.14.5%
126Gemini 3.5 FlashOpus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.14.5%
127GPT-6 Lunahigh effort · $0.0208 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.14.5%
128Qwen3.7 MaxQwen表:600题公开子集 · Pass@1· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.14.2%
129Claude Sonnet 5· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim13.5%
130Claude Sonnet 5Zapier private held-out set · max effort· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.13.5%
131GPT 5.5Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12.9%
132GPT 5.5OpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12.9%
133GPT 5.5Opus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12.9%
134GPT-5.6 Lunaxhigh effort · $0.06 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12.9%
135GPT-6 Lunaxhigh effort · $0.0245 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12.6%
136GPT-5.6 Sollow effort · $0.31 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.11.7%
137Claude Opus 4.7Opus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.9.9%
138Gemini 3.1 Pro PreviewFable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.9.6%
139Gemini 3.1 Pro PreviewOpus 4.8 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.9.6%
140GPT-6 Lunamedium effort · $0.0163 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.9.4%
141GPT-5.6 Lunahigh effort · $0.05 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.9.1%
142Claude Sonnet 4.6Zapier private held-out set · max effort· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.5.3%
143GPT-5.6 Lunamedium effort · $0.02 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.4.3%
144GPT-5.6 Lunalow effort · $0.01 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.1.8%
145GPT-6 Lunalow effort · $0.006 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.1.2%

Showing 101–145 of 145 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Fable 5.1 with Opus 5 fallbackAutomationBench v1.0.6 · 47 tools · OpenAI release chart · max reasoning · displayed cost $2.45/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.31.4%
Claude Fable 5.1 with Opus 5 fallbackmax effort · $2.45 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.31.4%
Claude Opus 5.5 with fallbacks (OpenAI chart label)AutomationBench v1.0.6 · 47 tools · OpenAI release chart · high reasoning · displayed cost $0.71/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.33%
Claude Opus 5.5 with fallbacks (OpenAI chart label)AutomationBench v1.0.6 · 47 tools · OpenAI release chart · low reasoning · displayed cost $0.51/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.24.2%
Claude Opus 5.5 with fallbacks (OpenAI chart label)AutomationBench v1.0.6 · 47 tools · OpenAI release chart · max reasoning · displayed cost $1.44/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.5%
Claude Opus 5.5 with fallbacks (OpenAI chart label)AutomationBench v1.0.6 · 47 tools · OpenAI release chart · medium reasoning · displayed cost $0.65/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.29.5%
Claude Opus 5.5 with fallbacks (OpenAI chart label)AutomationBench v1.0.6 · 47 tools · OpenAI release chart · xhigh reasoning · displayed cost $0.89/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.35.8%

Data credits: Data by LLM Stats · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.