Skip to content

Benchmark

BABILong avg(qa1-5) 128k

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models and systems measured
10
Scores recorded
10
Sources
1
IQ weight
No weight published

Score distribution

46.3%
Mean
45%
Median
28.2%
Std dev
13%
10th percentile
92.5%
90th percentile
7%
Min
7%96.9%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
~ ARMT (137M) fine-tuneavg, 128k· BABILong v0 data docs96.9%
~ Mamba (130M) fine-tuneavg, 128k· BABILong v0 data docs92.5%
~ RMT (137M) fine-tuneavg, 128k· BABILong v0 data docs58.5%
ChatGLM3 6Bchatglm3-6b-128k, avg, 128k· BABILong v0 data docs13%
GPT-4-0125-previewavg, 128k· BABILong v0 data docs36%
Llama 3.1 70B InstructMeta-Llama-3.1-70B-Instruct, avg, 128k· BABILong v0 data docs45%
Llama 3.1 8B InstructMeta-Llama-3.1-8B-Instruct, avg, 128k· BABILong v0 data docs39%
Llama3-ChatQA-1.5-8B + RAGavg, 128k· BABILong v0 data docs45%
Phi-3 Medium 128K InstructPhi-3-medium-128k-instruct, avg, 128k· BABILong v0 data docs30%
Phi-3 Mini 128K InstructPhi-3-mini-128k-instruct, avg, 128k· BABILong v0 data docs7%

Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.