Benchmark
BABILong qa2 4k
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
All long context benchmarksScores from BABILong v0 data docs.
- Models and systems measured
- 14
- Scores recorded
- 15
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 50.6%
- Mean
- 39%
- Median
- 27%
- Std dev
- 22%
- 10th percentile
- 98%
- 90th percentile
- 19%
- Min
Results
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model or system | Score |
|---|---|
| ~ ARMT (137M) fine-tuneqa2, 4k· BABILong v0 data docs | 100% |
| ~ Mamba (130M) fine-tuneqa2, 4k· BABILong v0 data docs | 98% |
| ~ RMT (137M) fine-tuneqa2, 4k· BABILong v0 data docs | 98% |
| ChatGLM3 6Bchatglm3-6b-128k, qa2, 4k, 128k· BABILong v0 data docs | 39% |
| GPT-4-0125-previewqa2, 4k· BABILong v0 data docs | 68% |
| Llama 2 7bLLaMA-2-7B-32K, qa2, 4k, 32k· BABILong v0 data docs | 26% |
| Llama 3 8B InstructMeta-Llama-3-8B-Instruct, qa2, 4k· BABILong v0 data docs | 39% |
| Llama3-ChatQA-1.5-8B + RAGqa2, 4k· BABILong v0 data docs | 19% |
| Mistral 7B Instruct v0.2Mistral-7b-Instruct-v0.2, qa2, 4k· BABILong v0 data docs | 22% |
| Mixtral 8x22B Instruct v0.1Mixtral-8x22B-Instruct-v0.1, qa2, 4k· BABILong v0 data docs | 56% |
| Mixtral 8x7B Instruct v0.1Mixtral-8x7B-Instruct-v0.1, qa2, 4k· BABILong v0 data docs | 35% |
| Phi-3 Medium 128K InstructPhi-3-medium-128k-instruct, qa2, 4k· BABILong v0 data docs | 51% |
| Phi-3 Mini 128K InstructPhi-3-mini-128k-instruct, qa2, 4k· BABILong v0 data docs | 36% |
| Yi-34B01-ai/Yi-34B-200k, qa2, 4k, 200k· BABILong v0 data docs | 42% |
| Yi-34B01-ai/Yi-34B, qa2, 4k· BABILong v0 data docs | 30% |
Data credits: RMT-team, Kuratov et al. Official v0 numeric CSV; results licence not separately stated.