Skip to content

Benchmark

BABILong qa4 128k

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models and systems measured
8
Scores recorded
8
Sources
1
IQ weight
No weight published

Score distribution

47.1%
Mean
38%
Median
33.8%
Std dev
11%
10th percentile
98%
90th percentile
5%
Min
5%100%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
~ ARMT (137M) fine-tuneqa4, 128k· BABILong v0 data docs100%
~ Mamba (130M) fine-tuneqa4, 128k· BABILong v0 data docs98%
~ RMT (137M) fine-tuneqa4, 128k· BABILong v0 data docs38%
ChatGLM3 6Bchatglm3-6b-128k, qa4, 128k· BABILong v0 data docs11%
GPT-4-0125-previewqa4, 128k· BABILong v0 data docs32%
Llama3-ChatQA-1.5-8B + RAGqa4, 128k· BABILong v0 data docs60%
Phi-3 Medium 128K InstructPhi-3-medium-128k-instruct, qa4, 128k· BABILong v0 data docs33%
Phi-3 Mini 128K InstructPhi-3-mini-128k-instruct, qa4, 128k· BABILong v0 data docs5%

Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.