Benchmark
RULER 128K
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
- Models measured
- 32
- Scores recorded
- 32
- Sources
- 3
- IQ weight
- 2.32
Score distribution
- 61.8%
- Mean
- 74.7%
- Median
- 30.1%
- Std dev
- 13.8%
- 10th percentile
- 94.4%
- 90th percentile
- 0%
- Min
This benchmark has a weight of 2.32 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Nemotron 3 Super 120B A12BNeMo Evaluator / mostly NeMo Skills · report Table 8 BF16 · dedicated RULER container· Reported by NVIDIA· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 97% |
| 2 | Gemma 4 31B ITGemma 4 Technical Report · Table 9 · without thinking · accuracy · 128K· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 96.4% |
| 3 | Gemma 4 26B A4BGemma 4 Technical Report · Table 9 · without thinking · accuracy · 128K· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it. | 89.8% |
| 4 | Granite 4.2 30B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim | 81.4% |
| 5 | Granite 4.2 8B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim | 71.4% |
| 6 | Granite 4.2 3B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim | 55.3% |
Not ranked
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model | Score |
|---|---|
| C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data | 74.7% |
| DBRX-instructDBRX, 36B/132B· RULER official results data | 0% |
| Gemini 1.5 ProGemini-1.5-pro· RULER official results data | 94.4% |
| GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data | 81.2% |
| Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 95.1% |
| Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data | 90% |
| Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data | 66.6% |
| Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data | 77% |
| Llama-2-7B-32K-InstructTogether, 7B· RULER official results data | 0% |
| Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data | 13.8% |
| Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data | 48.1% |
| Mistral NemoMistral-Nemo· RULER official results data | 19% |
| Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data | 23.7% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data | 31.7% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data | 44.5% |
| Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data | 46.1% |
| Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data | 66.7% |
| Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 90.6% |
| Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data | 0% |
| Qwen2 72B InstructQwen2, 72B· RULER official results data | 53.7% |
| Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 85.1% |
| Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 79.2% |
| Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 85.6% |
| Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 66% |
| Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it. | 77.4% |
| Yi-34BYi, 34B, 200k· RULER official results data | 77.3% |
Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted. · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links. · Data by LLM Stats