Skip to content

Benchmark

SWE-Bench Pro

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models and systems measured
93
Scores recorded
188
Sources
4
IQ weight
3.48

Score distribution

57.3%
Mean
57.7%
Median
14%
Std dev
46.3%
10th percentile
69.2%
90th percentile
5.2%
Min
5.2%97.7%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 3.48 for chat models, measured over 92 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#Model or systemScore
151Qwen3.6 27BQwen official release table · thinking mode unless benchmark requires otherwise · refined public set · internal scaffold · 200K· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.53.5%
152Claude Opus 4.6Claude Code · temp 1.0 · top_p 0.95 · 256K · refined tasks;Claude 对照为官方值 · max effort· Reported by Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.53.4%
153Mai Thinking 1· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim52.8%
154DeepSeek V4 Flash· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
52.6%
155DeepSeek V4 FlashDeepSeek V4 official model card · Think Max· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.52.6%
156GPT 5.4 Nano· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim52.4%
157GPT 5.4 Nanoxhigh · Public· Reported by OpenAI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.52.4%
158Muse Spark· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim52.4%
159DeepSeek V4 FlashDeepSeek V4 official model card · Think High· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.52.3%
160Claude Opus 4.5no extended thinking · 200K context · avg@5· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.52%
161NVIDIA Nemotron 3.5 Super VL 120B· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
51.8%
162Claude Opus 4.564K thinking budget · 200K context · avg@5· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.51.6%
163Muse Glimmer 30B· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
51.2%
164Muse Glimmer 30Bhigh reasoning· Reported by Meta· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.51.2%
165Kimi K2.5setting not named by the source· Reported by Moonshot AI· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.50.7%
166Kimi K2.5· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
50.7%
167Gemini 3 FlashGemini 3.5 Flash Model Card · May 2026 · Public · single attempt· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.49.6%
168Qwen3.6 35B A3B· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
49.5%
169Qwen3.6 FlashQwen3.6-35B-A3B official model card · hosted API name Qwen3.6-Flash· Reported by Alibaba Qwen· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.49.5%
170Laguna M.1· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim49.2%
171DeepSeek V4 FlashDeepSeek V4 official model card · Non-Think· Reported by DeepSeek· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.49.1%
172Laguna XS 2.1· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
47.6%
173Laguna XS.2· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim46.3%
174Qwen3 Coder Next· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
44.3%
175North Mini Code 1.0· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
40.2%
176Qwen3 Coder· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
38.7%
177MiniMax M2.1· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
36.8%
178Granite 4.2 30B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim33.3%
179Kimi K2 Instruct· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
27.7%
180Qwen 3 235B A22B· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
21.4%
181Granite 4.2 8B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim19.1%
182GPT OSS 120B· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
16.2%
183DeepSeek V3.2· OpenEvals Leaderboard data· Reported by the model's developer. The source relaying it did not verify it.15.6%
184Gemma 3 27B IT· OpenEvals Leaderboard data· Reported by the model's developer. The source relaying it did not verify it.11.4%
185Llama 3.1 405B Instruct· OpenEvals Leaderboard data· Reported by the model's developer. The source relaying it did not verify it.11.2%
186GLM 4.6· Hugging Face Official Benchmarks data· Reported by the model's developer. The source relaying it did not verify it. claim
Values from 2 sources
9.7%
187Llama 4 Maverick· OpenEvals Leaderboard data· Reported by the model's developer. The source relaying it did not verify it.5.2%

Showing 151–187 of 187 results.

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
Muse Spark (version unspecified)Meta Model API · highest effort · version unspecified · Scale AI · mini-swe-agent for Muse Spark 1.1· Reported by Meta, Meta Superintelligence Labs· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.55%

Data credits: Hugging Face official benchmark leaderboards; licence varies by benchmark · Data by LLM Stats · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links. · OpenEvals/leaderboard-data, MIT (licence)