Skip to content

Benchmark

RULER 64K

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models measured
29
Scores recorded
29
Sources
2
IQ weight
No weight published

Score distribution

73.8%
Mean
83.2%
Median
27.3%
Std dev
46.8%
10th percentile
93.3%
90th percentile
0%
Min
0%95.9%
The band covers the middle 80% of scores. The line marks the median.

Leaderboard

#ModelScore
1Granite 4.2 30B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim90%
2Granite 4.2 8B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim81%
3Granite 4.2 3B· LLM Stats data· Reported by the model's developer. The source relaying it did not verify it. claim67.5%

Not ranked

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

ModelScore
C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data86.6%
DBRX-instructDBRX, 36B/132B· RULER official results data2.4%
Gemini 1.5 ProGemini-1.5-pro· RULER official results data95.9%
GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data87%
Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.95.4%
Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data92.8%
Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data88.4%
Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data84.7%
Llama-2-7B-32K-InstructTogether, 7B· RULER official results data0%
Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data49%
Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data85.9%
Mistral NemoMistral-Nemo· RULER official results data46.8%
Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data78.8%
Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data84.7%
Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data72.4%
Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data78.6%
Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data80.6%
Qwen 3 235B A22BQwen3-235B-A22B, 235B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.93.3%
Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data0%
Qwen2 72B InstructQwen2, 72B· RULER official results data79.8%
Qwen3 14BQwen3-14B, 14B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.94%
Qwen3 30B A3BQwen3-30B-A3B, 30B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.89.1%
Qwen3 32BQwen3-32B, 32B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.91.8%
Qwen3 4BQwen3-4B, 4B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.77.8%
Qwen3 8BQwen3-8B, 8B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.82.1%
Yi-34BYi, 34B, 200k· RULER official results data83.2%

Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted. · Data by LLM Stats