Benchmark
NoCha pair accuracy (default)
Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.
- Models and systems measured
- 19
- Scores recorded
- 19
- Sources
- 1
- IQ weight
- No weight published
Score distribution
- 27.8%
- Mean
- 25%
- Median
- 16.6%
- Std dev
- 7.5%
- 10th percentile
- 49.4%
- 90th percentile
- 3.9%
- Min
Results
The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.
| Model or system | Score |
|---|---|
| BM25+GPT-4o (k=25)table: 3, view: default, system: BM25, GPT-4o (k=25)· NoCha (arXiv:2406.16264v3) data | 44.1% |
| BM25+GPT-4o (k=5)table: 3, view: default, system: BM25, GPT-4o (k=5)· NoCha (arXiv:2406.16264v3) data | 28.2% |
| BM25+GPT-4o (k=50)table: 3, view: default, system: BM25, GPT-4o (k=50)· NoCha (arXiv:2406.16264v3) data | 49.7% |
| Claude-3-Opustable: 3, view: default, system: Claude-3-Opus· NoCha (arXiv:2406.16264v3) data | 49.4% |
| Claude-3.5-Sonnettable: 3, view: default, system: Claude-3.5-Sonnet· NoCha (arXiv:2406.16264v3) data | 41% |
| Command Rtable: 3, view: default, system: Command R· NoCha (arXiv:2406.16264v3) data | 19.6% |
| Command R (simple prompt)table: 3, view: default, system: Command Rsimple· NoCha (arXiv:2406.16264v3) data | 22.5% |
| Command R+table: 3, view: default, system: Command R+· NoCha (arXiv:2406.16264v3) data | 17.3% |
| Command R+ (simple prompt)table: 3, view: default, system: Command R, simple· NoCha (arXiv:2406.16264v3) data | 13.7% |
| Gemini Flash 1.5table: 3, view: default, system: Gemini Flash 1.5· NoCha (arXiv:2406.16264v3) data | 34.2% |
| Gemini Pro 1.5table: 3, view: default, system: Gemini Pro 1.5· NoCha (arXiv:2406.16264v3) data | 48.1% |
| Gemma-10Mtable: 3, view: default, system: Gemma-10M· NoCha (arXiv:2406.16264v3) data | 3.9% |
| Gemma-10M (simple prompt)table: 3, view: default, system: Gemma-10Msimple· NoCha (arXiv:2406.16264v3) data | 7.5% |
| GPT-4-Turbotable: 3, view: default, system: GPT-4-Turbo· NoCha (arXiv:2406.16264v3) data | 40.2% |
| GPT-4otable: 3, view: default, system: GPT-4o· NoCha (arXiv:2406.16264v3) data | 55.8% |
| LongLLaMA (simple prompt)table: 3, view: default, system: LongLLaMAsimple· NoCha (arXiv:2406.16264v3) data | 4.9% |
| Phi-3-minitable: 3, view: default, system: Phi-3-mini· NoCha (arXiv:2406.16264v3) data | 9.3% |
| Phi-3-mini (simple prompt)table: 3, view: default, system: Phi-3-minisimple· NoCha (arXiv:2406.16264v3) data | 14.5% |
| Randomtable: 3, view: default, system: Random· NoCha (arXiv:2406.16264v3) data | 25% |
Data credits: Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal and Mohit Iyyer. One Thousand and One Pairs, arXiv:2406.16264v3. CC BY 4.0 article numeric/protocol extract; restructured, printed values unchanged. No task, book or dataset rights asserted. (licence)