Benchmark
BABILong avg(qa1-5) 4k
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
All long context benchmarksScores from BABILong v0 data docs.
- Models and systems measured
- 16
- Scores recorded
- 17
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 62.4%
- Mean
- 55%
- Median
- 18.8%
- Std dev
- 46%
- 10th percentile
- 92.3%
- 90th percentile
- 37%
- Min
Results
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model or system | Score |
|---|---|
| ~ ARMT (137M) fine-tuneavg, 4k· BABILong v0 data docs | 98.1% |
| ~ Mamba (130M) fine-tuneavg, 4k· BABILong v0 data docs | 98.7% |
| ~ RMT (137M) fine-tuneavg, 4k· BABILong v0 data docs | 92.3% |
| ChatGLM3 6Bchatglm3-6b-128k, avg, 4k, 128k· BABILong v0 data docs | 48% |
| GPT-4-0125-previewavg, 4k· BABILong v0 data docs | 74% |
| Llama 2 7bLLaMA-2-7B-32K, avg, 4k, 32k· BABILong v0 data docs | 40% |
| Llama 3 8B InstructMeta-Llama-3-8B-Instruct, avg, 4k· BABILong v0 data docs | 50% |
| Llama 3.1 70B InstructMeta-Llama-3.1-70B-Instruct, avg, 4k· BABILong v0 data docs | 74% |
| Llama 3.1 8B InstructMeta-Llama-3.1-8B-Instruct, avg, 4k· BABILong v0 data docs | 66% |
| Llama3-ChatQA-1.5-8B + RAGavg, 4k· BABILong v0 data docs | 46% |
| Mistral 7B Instruct v0.2Mistral-7b-Instruct-v0.2, avg, 4k· BABILong v0 data docs | 49% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B-Instruct-v0.1, avg, 4k· BABILong v0 data docs | 65% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B-Instruct-v0.1, avg, 4k· BABILong v0 data docs | 55% |
| Phi-3 Medium 128K InstructPhi-3-medium-128k-instruct, avg, 4k· BABILong v0 data docs | 62% |
| Phi-3 Mini 128K InstructPhi-3-mini-128k-instruct, avg, 4k· BABILong v0 data docs | 51% |
| Yi-34B01-ai/Yi-34B-200k, avg, 4k, 200k· BABILong v0 data docs | 54% |
| Yi-34B01-ai/Yi-34B, avg, 4k· BABILong v0 data docs | 37% |
Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.