Skip to content

Benchmark

SWE-bench-Live Lite

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
11
Scores recorded
22
Sources
1
IQ weight
1.09

Score distribution

27.2%
Mean
17.7%
Median
20.7%
Std dev
11.3%
10th percentile
63%
90th percentile
7%
Min
7%71%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.09 for chat models, measured over 11 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1DeepSeek V4.1 Flashscaffold: tianxicode, set: lite, instances: 300, submitted: 2026-10-08· SWE-bench-Live leaderboard data71%
2GPT-5.6 Solscaffold: slingshot-v3.4.0, set: lite, instances: 300, submitted: 2026-09-26· SWE-bench-Live leaderboard data70.3%
3Claude Opus 4.6scaffold: ami-agent, set: lite, instances: 300, submitted: 2026-06-23· SWE-bench-Live leaderboard data63%
4GPT 5.5scaffold: agav-agent, set: lite, instances: 300, submitted: 2026-09-03, effort: medium· SWE-bench-Live leaderboard data62%
5GPT-5.6 Solscaffold: slingshot-v3.3.0, set: lite, instances: 300, submitted: 2026-08-31· SWE-bench-Live leaderboard data44.7%
6Claude Sonnet 4.5scaffold: swe-agent, set: lite, instances: 300, submitted: 2025-12-01· SWE-bench-Live leaderboard data36%
7Claude Opus 4.8scaffold: aiwork.code, set: lite, instances: 300, submitted: 2026-09-09, effort: xhigh· SWE-bench-Live leaderboard data34%
8Claude Opus 4.6scaffold: slingshot-v3.1.0, set: lite, instances: 300, submitted: 2026-08-17· SWE-bench-Live leaderboard data25%
9GPT 5scaffold: swe-agent, set: lite, instances: 300, submitted: 2025-11-01, effort: medium· SWE-bench-Live leaderboard data24.3%
10Claude Sonnet 3.7scaffold: openhands, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data17.7%
11Claude Sonnet 3.7scaffold: swe-agent, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data17.7%
12GPT 4.1scaffold: swe-agent, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data16.3%
13DeepSeek V3scaffold: swe-agent, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data15.3%
14DeepSeek V3scaffold: agentless, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data13.3%
15DeepSeek V3scaffold: openhands, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data13%
16Claude Sonnet 4.5scaffold: slingshot-v2.6.0, set: lite, instances: 300, submitted: 2026-06-29· SWE-bench-Live leaderboard data12%
17GPT 4.1scaffold: agentless, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data12%
18GPT 4oscaffold: agentless, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data11.7%
19Claude Sonnet 3.7scaffold: agentless, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data11.3%
20GPT 4.1scaffold: openhands, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data11.3%
21GPT 4oscaffold: swe-agent, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data10%
22GPT 4oscaffold: openhands, set: lite, instances: 300, submitted: 2025-05-01· SWE-bench-Live leaderboard data7%

Data credits: SWE-bench-Live (Microsoft and contributors), swe-bench-live.github.io. Resolve rates computed from the published report counts.