Skip to content

Benchmark

SWE-bench-Live MultiLang Java

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
9
Scores recorded
23
Sources
1
IQ weight
1.14

Score distribution

37.4%
Mean
37.1%
Median
15.3%
Std dev
22.6%
10th percentile
67%
90th percentile
16.1%
Min
16.1%68.1%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.14 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1DeepSeek V4.1 Flashscaffold: tianxicode, set: java, instances: 160, submitted: 2026-10-16· SWE-bench-Live leaderboard data68.1%
2GPT-5.6 Solscaffold: slingshot-v3.4.0, set: java, instances: 160, submitted: 2026-10-08· SWE-bench-Live leaderboard data67.5%
3Claude Opus 4.6scaffold: ami-agent, set: java, instances: 109, submitted: 2026-08-17· SWE-bench-Live leaderboard data67%
4GPT-5.6 Solscaffold: slingshot-v3.3.0, set: java, instances: 160, submitted: 2026-09-01· SWE-bench-Live leaderboard data52.5%
5GPT 5.5scaffold: swe-agent, set: java, instances: 102, submitted: 2026-06-05, effort: medium· SWE-bench-Live leaderboard data50%
6GPT 5.5scaffold: openhands, set: java, instances: 102, submitted: 2026-06-05, effort: medium· SWE-bench-Live leaderboard data47.1%
7GPT 5.5scaffold: claude-code, set: java, instances: 102, submitted: 2026-06-05, effort: medium· SWE-bench-Live leaderboard data45.1%
8GPT 5.5scaffold: slingshot-v3.1.0, set: java, instances: 109, submitted: 2026-08-12, effort: medium· SWE-bench-Live leaderboard data41.3%
9DeepSeek V4 Proscaffold: openhands, set: java, instances: 102, submitted: 2026-06-05· SWE-bench-Live leaderboard data39.2%
10DeepSeek V4 Proscaffold: swe-agent, set: java, instances: 102, submitted: 2026-06-05· SWE-bench-Live leaderboard data39.2%
11DeepSeek V4 Proscaffold: claude-code, set: java, instances: 102, submitted: 2026-06-05· SWE-bench-Live leaderboard data38.2%
12GPT 5.2scaffold: claude-code, set: java, instances: 62, submitted: 2026-01-08, effort: medium, thinking· SWE-bench-Live leaderboard data37.1%
13GPT 5.5scaffold: slingshot-v2.6.0, set: java, instances: 109, submitted: 2026-07-07, effort: medium· SWE-bench-Live leaderboard data36.7%
14Claude Sonnet 4.5scaffold: claude-code, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data27.4%
15DeepSeek V3.1 Terminusscaffold: swe-agent, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data27.4%
16Gemini 3 Flashscaffold: swe-agent, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data25.8%
17DeepSeek V3.1 Terminusscaffold: openhands, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data24.2%
18Gemini 3 Flashscaffold: openhands, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data24.2%
19DeepSeek V3.1 Terminusscaffold: claude-code, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data22.6%
20GPT 5.2scaffold: openhands, set: java, instances: 62, submitted: 2026-01-08, effort: medium, thinking· SWE-bench-Live leaderboard data22.6%
21GPT 5.2scaffold: swe-agent, set: java, instances: 62, submitted: 2026-01-08, effort: medium, thinking· SWE-bench-Live leaderboard data22.6%
22Claude Sonnet 4.5scaffold: openhands, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data19.4%
23Claude Sonnet 4.5scaffold: swe-agent, set: java, instances: 62, submitted: 2026-01-08· SWE-bench-Live leaderboard data16.1%

Data credits: SWE-bench-Live (Microsoft and contributors), swe-bench-live.github.io. Resolve rates computed from the published report counts.