Benchmark
RULER 8K
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
All long context benchmarksScores from RULER official results data.
- Models measured
- 26
- Scores recorded
- 26
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 94%
- Mean
- 95.6%
- Median
- 3.5%
- Std dev
- 91.5%
- 10th percentile
- 97%
- 90th percentile
- 81.1%
- Min
Results
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model | Score |
|---|---|
| C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data | 93.7% |
| DBRX-instructDBRX, 36B/132B· RULER official results data | 93.8% |
| Gemini 1.5 ProGemini-1.5-pro· RULER official results data | 95.8% |
| GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data | 96.3% |
| Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 96.6% |
| Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data | 95.6% |
| Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data | 95.8% |
| Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data | 93.8% |
| Llama-2-7B-32K-InstructTogether, 7B· RULER official results data | 81.1% |
| Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data | 91.2% |
| Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data | 96.3% |
| Mistral NemoMistral-Nemo· RULER official results data | 87.2% |
| Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data | 96.1% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data | 94.9% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data | 92.1% |
| Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data | 93.2% |
| Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data | 91.5% |
| Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 97.2% |
| Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data | 93.8% |
| Qwen2 72B InstructQwen2, 72B· RULER official results data | 96.1% |
| Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 97.8% |
| Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 97% |
| Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 96% |
| Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 93.6% |
| Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 96% |
| Yi-34BYi, 34B, 200k· RULER official results data | 92.2% |
Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted.