Skip to content

Benchmark

Ramp SWE-Bench v2

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
6
Scores recorded
6
Sources
1
IQ weight
0.52

Score distribution

64.9%
Mean
64.4%
Median
8.1%
Std dev
59.6%
10th percentile
76.4%
90th percentile
52.4%
Min
52.4%76.4%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.52 for chat models, measured over 6 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

Data credits: Data by LLM Stats