Skip to content

Benchmark

RULER wAvg (inc)

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models measured
20
Scores recorded
20
Sources
1
IQ weight
No weight published

Score distribution

73.3%
Mean
79.6%
Median
18.9%
Std dev
38%
10th percentile
93.1%
90th percentile
33.8%
Min
33.8%95.7%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

ModelScore
C4AI Command R 08 2024Command-R-0824, 32B· RULER official results data86%
DBRX-instructDBRX, 36B/132B· RULER official results data38%
Gemini 1.5 ProGemini-1.5-pro· RULER official results data95.5%
GPT 4 1106 PreviewGPT-4-1106-preview· RULER official results data89%
Jamba 1.5 LargeJamba-1.5-large, 94B/398B· RULER official results data· Reported by the model's developer. The source relaying it did not verify it.95.7%
Jamba 1.5 MiniJamba-1.5-mini, 12B/52B· RULER official results data93.1%
Llama 3.1 70B InstructLlama3.1, 70B· RULER official results data85.5%
Llama 3.1 8B InstructLlama3.1, 8B· RULER official results data85.4%
Llama-2-7B-32K-InstructTogether, 7B· RULER official results data33.8%
Mistral 7B Instruct v0.2Mistral, 7B· RULER official results data55.6%
Mistral Large Instruct 2411Mistral-Large-2411, 123B· RULER official results data79.5%
Mistral NemoMistral-Nemo· RULER official results data54.7%
Mistral-Large-Instruct-2407Mistral-Large-2407, 123B· RULER official results data70.6%
Mixtral 8x22B Instruct v0.1Mixtral-8x22B, 39B/141B· RULER official results data73.5%
Mixtral 8x7B Instruct v0.1Mixtral-8x7B, 12.9B/46.7B· RULER official results data72.8%
Phi-3 Medium 128K InstructPhi3-medium, 14B· RULER official results data74.8%
Phi-3 Mini 128K InstructPhi3-mini, 3.8B· RULER official results data80.9%
Qwen1.5-72B-ChatQwen1.5, 72B· RULER official results data37.5%
Qwen2 72B InstructQwen2, 72B· RULER official results data79.6%
Yi-34BYi, 34B, 200k· RULER official results data84.8%

Data credits: NVIDIA RULER, Hsieh et al. Official README numeric table; no separate results licence asserted.