Skip to content

Benchmark

Vals SWE-bench Multimodal

Coding benchmarks compare models on programming tasks. Scores are percentages.

Models measured
21
Scores recorded
21
Sources
1
IQ weight
3.60

Score distribution

46.6%
Mean
43.5%
Median
9.4%
Std dev
37.5%
10th percentile
61.3%
90th percentile
33.1%
Min
33.1%70.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 3.60 for chat models, measured over 21 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard