Benchmark
RULER 32K
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
- Models measured
- 28
- Scores recorded
- 28
- Sources
- 2
- IQ weight
- No weight published
Score distribution
- 87.9%
- Mean
- 92.4%
- Median
- 10.2%
- Std dev
- 69%
- 10th percentile
- 96%
- 90th percentile
- 63%
- Min
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Gemma 4 26B A4BGemma 4 Technical Report · Table 9 · without thinking · accuracy · 32K· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 97.3% |
| 2 | Gemma 4 31B ITGemma 4 Technical Report · Table 9 · without thinking · accuracy · 32K· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 96.8% |
Not ranked
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model | Score |
|---|---|
| C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data | 90.8% |
| DBRX-instructDBRX, 36B/132B· RULER official results data | 63.1% |
| Gemini 1.5 ProGemini-1.5-pro· RULER official results data | 95.9% |
| GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data | 93.2% |
| Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 96% |
| Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data | 94.6% |
| Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data | 94.8% |
| Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data | 87.4% |
| Llama-2-7B-32K-InstructTogether, 7B· RULER official results data | 63% |
| Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data | 75.4% |
| Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data | 94% |
| Mistral NemoMistral-Nemo· RULER official results data | 69% |
| Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data | 93% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data | 90.9% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data | 85.9% |
| Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data | 86.8% |
| Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data | 87.5% |
| Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 95.1% |
| Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data | 67.8% |
| Qwen2 72B InstructQwen2, 72B· RULER official results data | 94.1% |
| Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 96.1% |
| Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 92.4% |
| Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 94.4% |
| Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 87.8% |
| Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 91.2% |
| Yi-34BYi, 34B, 200k· RULER official results data | 87.5% |
Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted. · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links.