Skip to content

Benchmark

Harvey Held-Out Legal Set

Scores are percentages. Higher is better.

Models measured
9
Scores recorded
11
Sources
1
IQ weight
0.17

Score distribution

15%
Mean
10%
Median
24.5%
Std dev
0.8%
10th percentile
13.3%
90th percentile
0%
Min
0%91.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.17 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude Sonnet 5Harvey held-out set · mean criterion pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.91.2%
2Claude Fable 5Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.13.3%
3Claude Opus 5Anthropic Opus 5 launch table · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.11.7%
4Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted · 120-task held-out set · high effort · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.11.7%
5Claude Opus 4.8Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.10.4%
6Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted · 120-task held-out set · max effort · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.10%
7Claude Opus 5.5Opus 5.5 System Card capability section · 120-task held-out set · max · Artificial Analysis harness · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.8.3%
8Claude Sonnet 5Harvey held-out set · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.5.8%
9GPT 5.5Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.2.1%
10Gemini 3.5 FlashFable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.0.8%
11Gemini 3.1 Pro PreviewFable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.0%

Data credits: BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.