Benchmark
Harvey Held-Out Legal Set
Scores are percentages. Higher is better.
Benchmark sourceScores from BenchBoard docs.
- Models measured
- 9
- Scores recorded
- 11
- Sources
- 1
- IQ weight
- 0.17
Score distribution
- 15%
- Mean
- 10%
- Median
- 24.5%
- Std dev
- 0.8%
- 10th percentile
- 13.3%
- 90th percentile
- 0%
- Min
This benchmark has a weight of 0.17 for chat models, measured over 9 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Claude Sonnet 5Harvey held-out set · mean criterion pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 91.2% |
| 2 | Claude Fable 5Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 13.3% |
| 3 | Claude Opus 5Anthropic Opus 5 launch table · adaptive thinking · max effort unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 11.7% |
| 4 | Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted · 120-task held-out set · high effort · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 11.7% |
| 5 | Claude Opus 4.8Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 10.4% |
| 6 | Claude Sonnet 5.5adaptive thinking · max effort · default sampling · 5-trial mean · context ≤1M unless noted · 120-task held-out set · max effort · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 10% |
| 7 | Claude Opus 5.5Opus 5.5 System Card capability section · 120-task held-out set · max · Artificial Analysis harness · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 8.3% |
| 8 | Claude Sonnet 5Harvey held-out set · all-pass· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 5.8% |
| 9 | GPT 5.5Fable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 2.1% |
| 10 | Gemini 3.5 FlashFable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 0.8% |
| 11 | Gemini 3.1 Pro PreviewFable/Mythos 5 System Card capability summary · adaptive thinking · max effort · 5 trials unless noted· Reported by Anthropic· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 0% |