Benchmark
DeepSWE v1.1
Scores are percentages. Higher is better.
Benchmark sourceScores from BenchBoard docs.
- Models and systems measured
- 46
- Scores recorded
- 151
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 61.7%
- Mean
- 67%
- Median
- 15.7%
- Std dev
- 42.4%
- 10th percentile
- 73.8%
- 90th percentile
- 1.2%
- Min
Leaderboard
| # | Model or system | Score |
|---|---|---|
| 101 | Claude Fable 5low effort · $3.7579 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59.6% |
| 102 | Muse Spark 1.2Muse Code· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59.3% |
| 103 | GPT-6 Lunahigh effort · $0.0838 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59.3% |
| 104 | Claude Opus 4.8max· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59% |
| 105 | Claude Opus 4.8max · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59% |
| 106 | Claude Opus 4.8OpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59% |
| 107 | Claude Opus 4.8Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59% |
| 108 | Claude Opus 4.8Qwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 59% |
| 109 | Qwen 3.8 Flash NextQwen3.8-Flash-Next release/model card · best of Claude Code and mini-SWE-agent · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58.7% |
| 110 | Qwen3.8 FlashClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58.7% |
| 111 | Claude Opus 5low effort · $1.66 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58.1% |
| 112 | Claude Opus 4.8mini-swe-agent / Z.ai设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 58% |
| 113 | Qwen3.8 Omni FlashClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 57.8% |
| 114 | GPT-6 Solmedium effort · $0.3798 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.6% |
| 115 | GPT-6 SolDeepSWE v1.1 · OpenAI release chart · medium reasoning · displayed cost $0.38/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.6% |
| 116 | Grok 4.5Grok Build· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.6% |
| 117 | Qwen3.8 Maxmini-swe-agent / Z.ai设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.6% |
| 118 | Qwen3.8 MaxQwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.6% |
| 119 | GPT-5.6 Lunaxhigh effort · $0.2737 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 56.2% |
| 120 | Muse Spark 1.2xhigh · official Datacurve leaderboard · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 55% |
| 121 | DeepSeek V4 Flash 0731Claude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54.4% |
| 122 | DeepSeek V4 Flash 0731DeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54.4% |
| 123 | Claude Sonnet 5Datacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54% |
| 124 | Claude Sonnet 5Gemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54% |
| 125 | Grok 4.5Gemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 54% |
| 126 | Muse Spark 1.1Meta Model API · xhigh · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 53.3% |
| 127 | Grok 4.5Datacurve mini-swe-agent harness· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 53% |
| 128 | Muse Spark 1.1mini-swe-agent· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 53% |
| 129 | Gemini 3.6 FlashGemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 49% |
| 130 | GLM 5.2mini-swe-agent · official pier framework · 400K context · 2h timeout· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 46.2% |
| 131 | GPT-5.6 Sollow effort · $0.82 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 45.4% |
| 132 | GPT-6 Lunamedium effort · $0.0518 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 44.5% |
| 133 | GLM 5.2Datacurve mini-swe-agent harness· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 44% |
| 134 | GPT-5.6 Lunahigh effort · $0.1272 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.4% |
| 135 | Qwen 3.8 27BClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.2% |
| 136 | Qwen 3.8 27BQwen3.8-27B model card · published setting· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 42.2% |
| 137 | Gemini 3.6 FlashAntigravity CLI · high· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 40% |
| 138 | GPT-6 SolDeepSWE v1.1 · OpenAI release chart · low reasoning · displayed cost $0.16/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 37.2% |
| 139 | GPT-6 Sollow effort · $0.1623 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 37.2% |
| 140 | Gemini 3.5 FlashGemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 37% |
| 141 | Qwen3.7 MaxQwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 21.6% |
| 142 | Qwen3.7 PlusClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 16.5% |
| 143 | Gemini 3.1 Pro PreviewGemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 12% |
| 144 | Gemini 3.1 Pro Previewhigh · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 12% |
| 145 | Gemini 3.1 Pro PreviewOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 11.8% |
| 146 | GPT-5.6 Lunamedium effort · $0.0313 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 9.3% |
| 147 | GPT-6 Lunalow effort · $0.0057 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 2.4% |
| 148 | GPT-5.6 Lunalow effort · $0.0106 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 1.2% |
Showing 101–148 of 148 results.
Not ranked
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model or system | Score |
|---|---|
| Claude Fable 5 with Opus 4.8 fallbackDatacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 69.7% |
| Grok 4.6Datacurve mini-SWE-agent · Pass@1 · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 65.2% |
| Muse Spark (version unspecified)Meta Model API · highest effort · version unspecified · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 10% |