Skip to content

Benchmark

BABILong avg(qa1-5) 1k

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models and systems measured
16
Scores recorded
17
Sources
1
IQ weight
No weight published

Score distribution

68.7%
Mean
63%
Median
16.1%
Std dev
53%
10th percentile
95.9%
90th percentile
48%
Min
48%98.6%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
~ ARMT (137M) fine-tuneavg, 1k· BABILong v0 data docs95.9%
~ Mamba (130M) fine-tuneavg, 1k· BABILong v0 data docs98.6%
~ RMT (137M) fine-tuneavg, 1k· BABILong v0 data docs97.4%
ChatGLM3 6Bchatglm3-6b-128k, avg, 1k, 128k· BABILong v0 data docs55%
GPT-4-0125-previewavg, 1k· BABILong v0 data docs81%
Llama 2 7bLLaMA-2-7B-32K, avg, 1k, 32k· BABILong v0 data docs53%
Llama 3 8B InstructMeta-Llama-3-8B-Instruct, avg, 1k· BABILong v0 data docs60%
Llama 3.1 70B InstructMeta-Llama-3.1-70B-Instruct, avg, 1k· BABILong v0 data docs81%
Llama 3.1 8B InstructMeta-Llama-3.1-8B-Instruct, avg, 1k· BABILong v0 data docs68%
Llama3-ChatQA-1.5-8B + RAGavg, 1k· BABILong v0 data docs48%
Mistral 7B Instruct v0.2Mistral-7b-Instruct-v0.2, avg, 1k· BABILong v0 data docs56%
Mixtral 8x22B Instruct v0.1Mixtral-8x22B-Instruct-v0.1, avg, 1k· BABILong v0 data docs73%
Mixtral 8x7B Instruct v0.1Mixtral-8x7B-Instruct-v0.1, avg, 1k· BABILong v0 data docs63%
Phi-3 Medium 128K InstructPhi-3-medium-128k-instruct, avg, 1k· BABILong v0 data docs70%
Phi-3 Mini 128K InstructPhi-3-mini-128k-instruct, avg, 1k· BABILong v0 data docs57%
Yi-34B01-ai/Yi-34B-200k, avg, 1k, 200k· BABILong v0 data docs59%
Yi-34B01-ai/Yi-34B, avg, 1k· BABILong v0 data docs52%

Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.