Benchmark
DeepSWE v1.1
Scores are percentages. Higher is better.
Benchmark sourceScores from BenchBoard docs.
- Models and systems measured
- 46
- Scores recorded
- 151
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 61.7%
- Mean
- 67%
- Median
- 15.7%
- Std dev
- 42.4%
- 10th percentile
- 73.8%
- 90th percentile
- 1.2%
- Min
Leaderboard
| # | Model or system | Score |
|---|---|---|
| 1 | Gemini 4 Argonofficial launch-page result· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 77.9% |
| 2 | DeepSeek V4.1 Flash 1017DeepSeek 1017 first-party evaluation · reasoning effort, candidate sampling, exact run count, and unspecified harnesses not published · Mini-SWE-V2 harness· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 77% |
| 3 | Muse Spark 1.3max · mini-swe-agent · 113 tasks · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 75.4% |
| 4 | GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · high reasoning · displayed cost $0.65/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 75.2% |
| 5 | Claude Opus 5.5Opus 5.5 System Card capability section · five-trial average· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74.2% |
| 6 | DeepSeek V4.1 FlashDeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74.2% |
| 7 | DeepSeek V4.1 Flashmini-SWE · max effort · temp 1.0 · top_p .95 · 1M · max_steps 500 · N=8· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74.2% |
| 8 | GPT-6 Astraxhigh effort · $4.4291 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74.1% |
| 9 | GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · xhigh reasoning · displayed cost $4.43/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74.1% |
| 10 | GPT-6 AstraOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74.1% |
| 11 | Claude Opus 5DeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74% |
| 12 | Claude Opus 5Google Model Card· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74% |
| 13 | Claude Opus 5max · official Datacurve leaderboard · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74% |
| 14 | Claude Opus 5mini-SWE / DeepSeek设置· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74% |
| 15 | GPT-6 Astrarounded prose result· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 74% |
| 16 | Gemini 3.8 FlashOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.8% |
| 17 | Claude Opus 5OpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.7% |
| 18 | Gemini 3.8 FlashGoogle Model Card· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.7% |
| 19 | MiniMax M3.1 FlashMiniMax M3.1 Flash official-weights model-card run · reasoning effort, sampling, and run count not stated for this benchmark · mini-SWE-agent 2.4.5 · 512K context· Reported by MiniMax· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.7% |
| 20 | Claude Opus 5max effort · $11.84 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.7% |
| 21 | GPT-6 Astrahigh effort · $3.9237 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.2% |
| 22 | GPT-6 Astramax effort · $7.4978 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.2% |
| 23 | GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · high reasoning · displayed cost $3.92/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.2% |
| 24 | GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · max reasoning · displayed cost $7.50/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.2% |
| 25 | Claude Opus 5xhigh effort · $9.07 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73.2% |
| 26 | GPT-5.6 SolDeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73% |
| 27 | GPT-5.6 Solmax · official Datacurve leaderboard · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73% |
| 28 | GPT-5.6 Solmini-SWE / DeepSeek设置· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73% |
| 29 | GPT-5.6 SolQwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73% |
| 30 | GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · medium reasoning · displayed cost $0.42/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 73% |
| 31 | Claude Opus 5high effort · $6.08 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.8% |
| 32 | GPT-6 AstraDeepSWE v1.1 · OpenAI release chart · medium reasoning · displayed cost $3.08/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.8% |
| 33 | GPT-6 Astramedium effort · $3.0755 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.8% |
| 34 | GPT-5.6 SolDatacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 35 | GPT-5.6 SolGoogle Model Card· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 36 | GPT-5.6 Solmini-swe-agent / Z.ai设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 37 | GPT-5.6 SolOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 38 | GPT-5.6 SolOpenAI launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 39 | GPT-5.6 SolOpus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 40 | GPT-5.6 Solmax effort · $6.46 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.7% |
| 41 | DeepSeek V4.1 FlashDSH Minimal · max effort · temp 1.0 · top_p .95 · 1M · max_steps 500 · N=8· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 72.6% |
| 42 | GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · max reasoning · displayed cost $1.57/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 71.9% |
| 43 | GPT-6.1 SolDeepSWE v1.1 · OpenAI release chart · xhigh reasoning · displayed cost $0.79/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 71.9% |
| 44 | Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted · five-trial average· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 71% |
| 45 | Grok 4.7Datacurve mini-SWE-agent · Pass@1 · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 71% |
| 46 | GPT-5.6 Solxhigh effort · $3.6 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 70.7% |
| 47 | DeepSeek V4.1 FlashDSH Standard · max effort · temp 1.0 · top_p .95 · 1M · max_steps 500 · N=8· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 70.5% |
| 48 | Claude Fable 5max· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 70% |
| 49 | Claude Fable 5Qwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 70% |
| 50 | Claude Fable 5xhigh effort · $13.4145 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 69.9% |
Showing 1–50 of 148 results.
Not ranked
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model or system | Score |
|---|---|
| Claude Fable 5 with Opus 4.8 fallbackDatacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 69.7% |
| Grok 4.6Datacurve mini-SWE-agent · Pass@1 · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 65.2% |
| Muse Spark (version unspecified)Meta Model API · highest effort · version unspecified · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 10% |