Benchmark
RULER 64K
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
All long context benchmarksScores from RULER official results data, LLM Stats data.
- Models measured
- 29
- Scores recorded
- 29
- Sources
- 2
- IQ weight
- No weight published
Score distribution
- 73.8%
- Mean
- 83.2%
- Median
- 27.3%
- Std dev
- 46.8%
- 10th percentile
- 93.3%
- 90th percentile
- 0%
- Min
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Granite 4.2 30B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim | 90% |
| 2 | Granite 4.2 8B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim | 81% |
| 3 | Granite 4.2 3B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim | 67.5% |
Not ranked
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model | Score |
|---|---|
| C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data | 86.6% |
| DBRX-instructDBRX, 36B/132B· RULER official results data | 2.4% |
| Gemini 1.5 ProGemini-1.5-pro· RULER official results data | 95.9% |
| GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data | 87% |
| Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 95.4% |
| Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data | 92.8% |
| Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data | 88.4% |
| Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data | 84.7% |
| Llama-2-7B-32K-InstructTogether, 7B· RULER official results data | 0% |
| Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data | 49% |
| Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data | 85.9% |
| Mistral NemoMistral-Nemo· RULER official results data | 46.8% |
| Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data | 78.8% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data | 84.7% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data | 72.4% |
| Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data | 78.6% |
| Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data | 80.6% |
| Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 93.3% |
| Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data | 0% |
| Qwen2 72B InstructQwen2, 72B· RULER official results data | 79.8% |
| Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 94% |
| Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 89.1% |
| Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 91.8% |
| Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 77.8% |
| Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 82.1% |
| Yi-34BYi, 34B, 200k· RULER official results data | 83.2% |
Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted. · Data by LLM Stats