Skip to content

Benchmark

DeepSWE v1.1

Scores are percentages. Higher is better.

Models and systems measured
46
Scores recorded
151
Sources
1
IQ weight
No weight published

Score distribution

61.7%
Mean
67%
Median
15.7%
Std dev
42.4%
10th percentile
73.8%
90th percentile
1.2%
Min
1.2%77.9%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#Model or systemScore
101Claude Fable 5low effort · $3.7579 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59.6%
102Muse Spark 1.2Muse Code· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59.3%
103GPT-6 Lunahigh effort · $0.0838 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59.3%
104Claude Opus 4.8max· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59%
105Claude Opus 4.8max · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59%
106Claude Opus 4.8OpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59%
107Claude Opus 4.8Opus 5 System Card capability summary · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59%
108Claude Opus 4.8Qwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.59%
109Qwen 3.8 Flash NextQwen3.8-Flash-Next release/model card · best of Claude Code and mini-SWE-agent · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58.7%
110Qwen3.8 FlashClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58.7%
111Claude Opus 5low effort · $1.66 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58.1%
112Claude Opus 4.8mini-swe-agent / Z.ai设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.58%
113Qwen3.8 Omni FlashClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.57.8%
114GPT-6 Solmedium effort · $0.3798 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.6%
115GPT-6 SolDeepSWE v1.1 · OpenAI release chart · medium reasoning · displayed cost $0.38/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.6%
116Grok 4.5Grok Build· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.6%
117Qwen3.8 Maxmini-swe-agent / Z.ai设置· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.6%
118Qwen3.8 MaxQwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.6%
119GPT-5.6 Lunaxhigh effort · $0.2737 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.56.2%
120Muse Spark 1.2xhigh · official Datacurve leaderboard · task pass rate· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.55%
121DeepSeek V4 Flash 0731Claude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.4%
122DeepSeek V4 Flash 0731DeepSeek V4.1 launch chart · exact public value · setting not further specified· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54.4%
123Claude Sonnet 5Datacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54%
124Claude Sonnet 5Gemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54%
125Grok 4.5Gemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.54%
126Muse Spark 1.1Meta Model API · xhigh · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.53.3%
127Grok 4.5Datacurve mini-swe-agent harness· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.53%
128Muse Spark 1.1mini-swe-agent· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.53%
129Gemini 3.6 FlashGemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.49%
130GLM 5.2mini-swe-agent · official pier framework · 400K context · 2h timeout· Reported by Z.ai· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.46.2%
131GPT-5.6 Sollow effort · $0.82 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.45.4%
132GPT-6 Lunamedium effort · $0.0518 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.44.5%
133GLM 5.2Datacurve mini-swe-agent harness· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.44%
134GPT-5.6 Lunahigh effort · $0.1272 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.4%
135Qwen 3.8 27BClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.2%
136Qwen 3.8 27BQwen3.8-27B model card · published setting· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.42.2%
137Gemini 3.6 FlashAntigravity CLI · high· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.40%
138GPT-6 SolDeepSWE v1.1 · OpenAI release chart · low reasoning · displayed cost $0.16/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.37.2%
139GPT-6 Sollow effort · $0.1623 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.37.2%
140Gemini 3.5 FlashGemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.37%
141Qwen3.7 MaxQwen表:Claude Code / mini-SWE-agent 取较高值 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.21.6%
142Qwen3.7 PlusClaude Code / mini-SWE-agent 取较高值 · temp 1.0 · top_p 0.95 · 256K· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.16.5%
143Gemini 3.1 Pro PreviewGemini 3.6 Flash Model Card · July 2026· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12%
144Gemini 3.1 Pro Previewhigh · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.12%
145Gemini 3.1 Pro PreviewOpenAI GPT-5.6 launch table· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.11.8%
146GPT-5.6 Lunamedium effort · $0.0313 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.9.3%
147GPT-6 Lunalow effort · $0.0057 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.2.4%
148GPT-5.6 Lunalow effort · $0.0106 average cost/task· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.1.2%

Showing 101–148 of 148 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Claude Fable 5 with Opus 4.8 fallbackDatacurve mini-SWE-agent · Pass@1 · max effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.69.7%
Grok 4.6Datacurve mini-SWE-agent · Pass@1 · high effort· Reported by SpaceXAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.65.2%
Muse Spark (version unspecified)Meta Model API · highest effort · version unspecified · mini-swe-agent fork · no internet · 64GB VM · pass@1 · five-attempt mean· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.10%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.