Skip to content

Benchmark

BABILong qa2 64k

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models and systems measured
10
Scores recorded
10
Sources
1
IQ weight
No weight published

Score distribution

36.2%
Mean
25%
Median
34%
Std dev
7%
10th percentile
95%
90th percentile
6%
Min
6%100%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
~ ARMT (137M) fine-tuneqa2, 64k· BABILong v0 data docs100%
~ Mamba (130M) fine-tuneqa2, 64k· BABILong v0 data docs95%
~ RMT (137M) fine-tuneqa2, 64k· BABILong v0 data docs59%
ChatGLM3 6Bchatglm3-6b-128k, qa2, 64k, 128k· BABILong v0 data docs6%
GPT-4-0125-previewqa2, 64k· BABILong v0 data docs25%
Llama3-ChatQA-1.5-8B + RAGqa2, 64k· BABILong v0 data docs7%
Mixtral 8x22B Instruct v0.1Mixtral-8x22B-Instruct-v0.1, qa2, 64k· BABILong v0 data docs11%
Phi-3 Medium 128K InstructPhi-3-medium-128k-instruct, qa2, 64k· BABILong v0 data docs14%
Phi-3 Mini 128K InstructPhi-3-mini-128k-instruct, qa2, 64k· BABILong v0 data docs15%
Yi-34B01-ai/Yi-34B-200k, qa2, 64k, 200k· BABILong v0 data docs30%

Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.