Skip to content

Benchmark

RULER wAvg (dec)

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models measured
20
Scores recorded
20
Sources
1
IQ weight
No weight published

Score distribution

87.7%
Mean
90.4%
Median
8.1%
Std dev
74.7%
10th percentile
94.8%
90th percentile
66.7%
Min
66.7%96.3%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

ModelScore
C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data91.9%
DBRX-instructDBRX, 36B/132B· RULER official results data74.7%
Gemini 1.5 ProGemini-1.5-pro· RULER official results data96.1%
GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data94.1%
Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.96.3%
Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data94.8%
Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data93.7%
Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data91.3%
Llama-2-7B-32K-InstructTogether, 7B· RULER official results data66.7%
Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data81.2%
Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data92.5%
Mistral NemoMistral-Nemo· RULER official results data77.8%
Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data90.4%
Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data90.3%
Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data87.9%
Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data88.3%
Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data88.7%
Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data74%
Qwen2 72B InstructQwen2, 72B· RULER official results data92.3%
Yi-34BYi, 34B, 200k· RULER official results data90.1%

Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted.