Skip to content

Benchmark

RULER 128K

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models measured
32
Scores recorded
32
Sources
3
IQ weight
2.32

Score distribution

61.8%
Mean
74.7%
Median
30.1%
Std dev
13.8%
10th percentile
94.4%
90th percentile
0%
Min
0%97%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 2.32 for chat models, measured over 5 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Nemotron 3 Super 120B A12BNeMo Evaluator / mostly NeMo Skills · report Table 8 BF16 · dedicated RULER container· Reported by NVIDIA· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.97%
2Gemma 4 31B ITGemma 4 Technical Report · Table 9 · without thinking · accuracy · 128K· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.96.4%
3Gemma 4 26B A4BGemma 4 Technical Report · Table 9 · without thinking · accuracy · 128K· Reported by Google DeepMind· via BenchBoard docs· Reported by the model's developer. The source relaying it did not verify it.89.8%
4Granite 4.2 30B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim81.4%
5Granite 4.2 8B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim71.4%
6Granite 4.2 3B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim55.3%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

ModelScore
C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data74.7%
DBRX-instructDBRX, 36B/132B· RULER official results data0%
Gemini 1.5 ProGemini-1.5-pro· RULER official results data94.4%
GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data81.2%
Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.95.1%
Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data90%
Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data66.6%
Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data77%
Llama-2-7B-32K-InstructTogether, 7B· RULER official results data0%
Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data13.8%
Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data48.1%
Mistral NemoMistral-Nemo· RULER official results data19%
Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data23.7%
Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data31.7%
Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data44.5%
Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data46.1%
Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data66.7%
Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.90.6%
Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data0%
Qwen2 72B InstructQwen2, 72B· RULER official results data53.7%
Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.85.1%
Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.79.2%
Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.85.6%
Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.66%
Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.77.4%
Yi-34BYi, 34B, 200k· RULER official results data77.3%

Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted. · BenchBoard by SYuan03 (github.com/SYuan03/benchboard). MIT covers its code and data organization; no results licence is inferred. Numeric relays retain the original stating-source links. · Data by LLM Stats