Skip to content

Benchmark

BABILong qa1 512k

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models and systems measured
4
Scores recorded
4
Sources
1
IQ weight
No weight published

Score distribution

83%
Mean
92%
Median
17.3%
Std dev
54%
10th percentile
99%
90th percentile
54%
Min
54%99%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
~ ARMT (137M) fine-tuneqa1, 512k· BABILong v0 data docs99%
~ Mamba (130M) fine-tuneqa1, 512k· BABILong v0 data docs92%
~ RMT (137M) fine-tuneqa1, 512k· BABILong v0 data docs87%
Llama3-ChatQA-1.5-8B + RAGqa1, 512k· BABILong v0 data docs54%

Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.