Benchmark
RULER Avg
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
All long context benchmarksScores from RULER official results data.
- Models measured
- 26
- Scores recorded
- 26
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 83%
- Mean
- 87.5%
- Median
- 12.7%
- Std dev
- 66.2%
- 10th percentile
- 95%
- 90th percentile
- 50.3%
- Min
Results
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model | Score |
|---|---|
| C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data | 88.9% |
| DBRX-instructDBRX, 36B/132B· RULER official results data | 56.3% |
| Gemini 1.5 ProGemini-1.5-pro· RULER official results data | 95.8% |
| GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data | 91.6% |
| Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 96% |
| Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data | 93.9% |
| Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data | 89.6% |
| Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data | 88.3% |
| Llama-2-7B-32K-InstructTogether, 7B· RULER official results data | 50.3% |
| Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data | 68.4% |
| Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data | 86% |
| Mistral NemoMistral-Nemo· RULER official results data | 66.2% |
| Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data | 80.5% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data | 81.9% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data | 80.4% |
| Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data | 81.5% |
| Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data | 84.8% |
| Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 95% |
| Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data | 55.7% |
| Qwen2 72B InstructQwen2, 72B· RULER official results data | 85.9% |
| Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 94.6% |
| Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 91.6% |
| Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 93.7% |
| Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 85.2% |
| Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 89.1% |
| Yi-34BYi, 34B, 200k· RULER official results data | 87.5% |
Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted.