Skip to content

Benchmark

NoCha pair accuracy (default)

Long context benchmarks measure how well models use information in long inputs. Scores are percentages. Higher is better.

Models and systems measured
19
Scores recorded
19
Sources
1
IQ weight
No weight published

Score distribution

27.8%
Mean
25%
Median
16.6%
Std dev
7.5%
10th percentile
49.4%
90th percentile
3.9%
Min
3.9%55.8%
The band covers the middle 80% of scores. The line marks the median.

Results

The source marks these results as descriptive, not capability scores, so we list them by name instead of ranking them.

Model or systemScore
BM25+GPT-4o (k=25)table: 3, view: default, system: BM25, GPT-4o (k=25)· NoCha (arXiv:2406.16264v3) data44.1%
BM25+GPT-4o (k=5)table: 3, view: default, system: BM25, GPT-4o (k=5)· NoCha (arXiv:2406.16264v3) data28.2%
BM25+GPT-4o (k=50)table: 3, view: default, system: BM25, GPT-4o (k=50)· NoCha (arXiv:2406.16264v3) data49.7%
Claude-3-Opustable: 3, view: default, system: Claude-3-Opus· NoCha (arXiv:2406.16264v3) data49.4%
Claude-3.5-Sonnettable: 3, view: default, system: Claude-3.5-Sonnet· NoCha (arXiv:2406.16264v3) data41%
Command Rtable: 3, view: default, system: Command R· NoCha (arXiv:2406.16264v3) data19.6%
Command R (simple prompt)table: 3, view: default, system: Command Rsimple· NoCha (arXiv:2406.16264v3) data22.5%
Command R+table: 3, view: default, system: Command R+· NoCha (arXiv:2406.16264v3) data17.3%
Command R+ (simple prompt)table: 3, view: default, system: Command R, simple· NoCha (arXiv:2406.16264v3) data13.7%
Gemini Flash 1.5table: 3, view: default, system: Gemini Flash 1.5· NoCha (arXiv:2406.16264v3) data34.2%
Gemini Pro 1.5table: 3, view: default, system: Gemini Pro 1.5· NoCha (arXiv:2406.16264v3) data48.1%
Gemma-10Mtable: 3, view: default, system: Gemma-10M· NoCha (arXiv:2406.16264v3) data3.9%
Gemma-10M (simple prompt)table: 3, view: default, system: Gemma-10Msimple· NoCha (arXiv:2406.16264v3) data7.5%
GPT-4-Turbotable: 3, view: default, system: GPT-4-Turbo· NoCha (arXiv:2406.16264v3) data40.2%
GPT-4otable: 3, view: default, system: GPT-4o· NoCha (arXiv:2406.16264v3) data55.8%
LongLLaMA (simple prompt)table: 3, view: default, system: LongLLaMAsimple· NoCha (arXiv:2406.16264v3) data4.9%
Phi-3-minitable: 3, view: default, system: Phi-3-mini· NoCha (arXiv:2406.16264v3) data9.3%
Phi-3-mini (simple prompt)table: 3, view: default, system: Phi-3-minisimple· NoCha (arXiv:2406.16264v3) data14.5%
Randomtable: 3, view: default, system: Random· NoCha (arXiv:2406.16264v3) data25%

Data credits: Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal and Mohit Iyyer. One Thousand and One Pairs, arXiv:2406.16264v3. CC BY 4.0 article numeric/protocol extract; restructured, printed values unchanged. No task, book or dataset rights asserted. (licence)