Skip to content

Benchmark

DeepSWE v1.1

Scores are percentages. Higher is better.

Models and systems measured
46
Scores recorded
151
Sources
1
IQ weight
No weight published

Score distribution

61.7%
Mean
67%
Median
15.7%
Std dev
42.4%
10th percentile
73.8%
90th percentile
1.2%
Min
1.2%77.9%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#Model or systemScore
1Gemini 4 Argonofficial launch-page result· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.77.9%
2DeepSeek V4.1 Flash 1017DeepSeek 1017 first-party evaluation · reasoning effort, candidate sampling, exact run count, and unspecified harnesses not published · Mini-SWE-V2 harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.77%
3Muse Spark 1.3max · mini-swe-agent · 113 tasks · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.75.4%
4GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · high reasoning · displayed cost $0.65/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.75.2%
5Claude Opus 5.5Opus 5.5 System Card capability section · five-trial average· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74.2%
6DeepSeek V4.1 FlashDeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74.2%
7DeepSeek V4.1 Flashmini-SWE · max effort · temp 1.0 · top_p .95 · 1M · max_steps 500 · N=8· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74.2%
8GPT-6 Astraxhigh effort · $4.4291 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74.1%
9GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · xhigh reasoning · displayed cost $4.43/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74.1%
10GPT-6 AstraOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74.1%
11Claude Opus 5DeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74%
12Claude Opus 5Google Model Card· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74%
13Claude Opus 5max · official Datacurve leaderboard · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74%
14Claude Opus 5mini-SWE / DeepSeek设置· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74%
15GPT-6 Astrarounded prose result· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.74%
16Gemini 3.8 FlashOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.8%
17Claude Opus 5OpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.7%
18Gemini 3.8 FlashGoogle Model Card· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.7%
19MiniMax M3.1 FlashMiniMax M3.1 Flash official-weights model-card run · reasoning effort, sampling, and run count not stated for this benchmark · mini-SWE-agent 2.4.5 · 512K context· Reported by MiniMax· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.7%
20Claude Opus 5max effort · $11.84 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.7%
21GPT-6 Astrahigh effort · $3.9237 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.2%
22GPT-6 Astramax effort · $7.4978 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.2%
23GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · high reasoning · displayed cost $3.92/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.2%
24GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · max reasoning · displayed cost $7.50/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.2%
25Claude Opus 5xhigh effort · $9.07 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73.2%
26GPT-5.6 SolDeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73%
27GPT-5.6 Solmax · official Datacurve leaderboard · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73%
28GPT-5.6 Solmini-SWE / DeepSeek设置· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73%
29GPT-5.6 SolQwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73%
30GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · medium reasoning · displayed cost $0.42/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.73%
31Claude Opus 5high effort · $6.08 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.8%
32GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · medium reasoning · displayed cost $3.08/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.8%
33GPT-6 Astramedium effort · $3.0755 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.8%
34GPT-5.6 SolDatacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
35GPT-5.6 SolGoogle Model Card· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
36GPT-5.6 Solmini-swe-agent / Z.ai设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
37GPT-5.6 SolOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
38GPT-5.6 SolOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
39GPT-5.6 SolOpus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
40GPT-5.6 Solmax effort · $6.46 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.7%
41DeepSeek V4.1 FlashDSH Minimal · max effort · temp 1.0 · top_p .95 · 1M · max_steps 500 · N=8· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.72.6%
42GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · max reasoning · displayed cost $1.57/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.71.9%
43GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · xhigh reasoning · displayed cost $0.79/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.71.9%
44Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted · five-trial average· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.71%
45Grok 4.7Datacurve mini-SWE-agent · Pass@1 · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.71%
46GPT-5.6 Solxhigh effort · $3.6 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.70.7%
47DeepSeek V4.1 FlashDSH Standard · max effort · temp 1.0 · top_p .95 · 1M · max_steps 500 · N=8· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.70.5%
48Claude Fable 5max· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.70%
49Claude Fable 5Qwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.70%
50Claude Fable 5xhigh effort · $13.4145 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.69.9%

Showing 1–50 of 148 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Fable 5 with Opus 4.8 fallbackDatacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.69.7%
Grok 4.6Datacurve mini-SWE-agent · Pass@1 · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.65.2%
Muse Spark (version unspecified)Meta Model API · highest effort · version unspecified · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.10%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.