Skip to content

Benchmark

SWE-bench-Live MultiLang Rust

Coding benchmarks compare models on programming tasks. Scores are percentages. Higher is better.

Models measured
8
Scores recorded
19
Sources
1
IQ weight
0.93

Score distribution

31.4%
Mean
28.9%
Median
7.8%
Std dev
26.7%
10th percentile
37.8%
90th percentile
17.8%
Min
17.8%50.9%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.93 for chat models, measured over 8 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1DeepSeek V4.1 Flashscaffold: tianxicode, set: rust, instances: 171, submitted: 2026-10-20· SWE-bench-Live leaderboard data50.9%
2Claude Opus 4.6scaffold: ami-agent, set: rust, instances: 94, submitted: 2026-06-25· SWE-bench-Live leaderboard data48.9%
3Gemini 3 Flashscaffold: swe-agent, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data37.8%
4Gemini 3 Flashscaffold: openhands, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data35.6%
5DeepSeek V4 Proscaffold: swe-agent, set: rust, instances: 64, submitted: 2026-06-05· SWE-bench-Live leaderboard data34.4%
6GPT 5.2scaffold: claude-code, set: rust, instances: 45, submitted: 2026-01-08, effort: medium, thinking· SWE-bench-Live leaderboard data33.3%
7GPT 5.5scaffold: claude-code, set: rust, instances: 64, submitted: 2026-06-05, effort: medium· SWE-bench-Live leaderboard data32.8%
8DeepSeek V4 Proscaffold: claude-code, set: rust, instances: 64, submitted: 2026-06-05· SWE-bench-Live leaderboard data29.7%
9GPT 5.5scaffold: openhands, set: rust, instances: 64, submitted: 2026-06-05, effort: medium· SWE-bench-Live leaderboard data29.7%
10Claude Sonnet 4.5scaffold: swe-agent, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data28.9%
11DeepSeek V3.1 Terminusscaffold: swe-agent, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data28.9%
12GPT 5.2scaffold: swe-agent, set: rust, instances: 45, submitted: 2026-01-08, effort: medium, thinking· SWE-bench-Live leaderboard data28.9%
13DeepSeek V4 Proscaffold: openhands, set: rust, instances: 64, submitted: 2026-06-05· SWE-bench-Live leaderboard data28.1%
14GPT 5.5scaffold: swe-agent, set: rust, instances: 64, submitted: 2026-06-05, effort: medium· SWE-bench-Live leaderboard data28.1%
15Claude Sonnet 4.5scaffold: openhands, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data26.7%
16DeepSeek V3.1 Terminusscaffold: openhands, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data26.7%
17GPT 5.2scaffold: openhands, set: rust, instances: 45, submitted: 2026-01-08, effort: medium, thinking· SWE-bench-Live leaderboard data26.7%
18Claude Sonnet 4.5scaffold: claude-code, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data22.2%
19DeepSeek V3.1 Terminusscaffold: claude-code, set: rust, instances: 45, submitted: 2026-01-08· SWE-bench-Live leaderboard data17.8%

Data credits: SWE-bench-Live (Microsoft and contributors), swe-bench-live.github.io. Resolve rates computed from the published report counts.