Skip to content

Benchmark

FictionLiveBench 32k

Long context benchmarks measure how well models use information in long inputs. Scores are percentages.

Models measured
47
Scores recorded
50
Sources
1
IQ weight
1.93

Score distribution

59.1%
Mean
55.6%
Median
22.5%
Std dev
33.3%
10th percentile
91.7%
90th percentile
5%
Min
5%100%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.93 for chat models, measured over 47 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1O3 Promedium· Epoch AI Benchmarking Hub data100%
2OpenAI o3medium· Epoch AI Benchmarking Hub data100%
3GPT 5medium· Epoch AI Benchmarking Hub data97.2%
4Gemini 2.5 Pro· Epoch AI Benchmarking Hub data91.7%
5Grok 4· Epoch AI Benchmarking Hub data91.7%
6Grok 4 Fast· Epoch AI Benchmarking Hub data91.7%
7Kimi K2.5· Epoch AI Benchmarking Hub data88.9%
8Gemini 2.5 Pro· Epoch AI Benchmarking Hub data86.1%
9DeepSeek V3.2 Exphigh· Epoch AI Benchmarking Hub data83.3%
10o1medium· Epoch AI Benchmarking Hub data83.3%
11Qwen 3 235B A22B Thinking 2507· Epoch AI Benchmarking Hub data80.6%
12QwQ 32B· Epoch AI Benchmarking Hub data80.6%
13Claude Sonnet 3.78k· Epoch AI Benchmarking Hub data75%
14Grok 3 Minimedium, beta· Epoch AI Benchmarking Hub data75%
15Gemini 2.5 Pro Preview 05-06· Epoch AI Benchmarking Hub data72.2%
16MiniMax M1 80K· Epoch AI Benchmarking Hub data72.2%
17DeepSeek R1· Epoch AI Benchmarking Hub data69.4%
18GLM 4.5· Epoch AI Benchmarking Hub data66.7%
19Claude Opus 4· Epoch AI Benchmarking Hub data63.9%
20GPT-4.5 Preview· Epoch AI Benchmarking Hub data63.9%
21GPT-5 minimedium· Epoch AI Benchmarking Hub data63.9%
22Kimi K2 Instruct· Epoch AI Benchmarking Hub data63.9%
23GPT 4.1· Epoch AI Benchmarking Hub data58.3%
24DeepSeek V3· Epoch AI Benchmarking Hub data55.6%
25Gemini 2.5 Flash Preview 05-20 (no think)· Epoch AI Benchmarking Hub data55.6%
26Grok 3beta· Epoch AI Benchmarking Hub data55.6%
27o3 Minimedium· Epoch AI Benchmarking Hub data55.6%
28o4 Minimedium· Epoch AI Benchmarking Hub data55.6%
29Qwen3 235B A22B Instruct 2507· Epoch AI Benchmarking Hub data52.9%
30Claude Sonnet 3.7· Epoch AI Benchmarking Hub data52.8%
31Qwen3 Next 80B A3B Instruct· Epoch AI Benchmarking Hub data52.8%
32DeepSeek V3.2 Exp· Epoch AI Benchmarking Hub data50%
33Gemini 2.0 Flash· Epoch AI Benchmarking Hub data50%
34Gemini 2.5 Pro Preview 03-25· Epoch AI Benchmarking Hub data50%
35gemini-2.0-pro-exp-02-05· Epoch AI Benchmarking Hub data47.2%
36GPT-5 nanomedium· Epoch AI Benchmarking Hub data47.2%
37Claude Sonnet 4· Epoch AI Benchmarking Hub data44.4%
38Gemini 2.5 Flash-Litethinking· Epoch AI Benchmarking Hub data44.4%
39gemini-2.5-flash-preview-04-17· Epoch AI Benchmarking Hub data44.4%
40Llama 4 Maverick· Epoch AI Benchmarking Hub data44%
41GPT OSS 120Bhigh· Epoch AI Benchmarking Hub data38.9%
42GPT-4.1 mini· Epoch AI Benchmarking Hub data38.9%
43Gemini 2.0 Flash Experimentalthinking· Epoch AI Benchmarking Hub data36.1%
44GPT OSS 120B· Epoch AI Benchmarking Hub data33.3%
45GPT-4.1 nano· Epoch AI Benchmarking Hub data33.3%
46Llama 3.3 70B Instruct· Epoch AI Benchmarking Hub data33.3%
47Gemma 3 27B IT· Epoch AI Benchmarking Hub data25%
48Nemotron Nano 9b V2· Epoch AI Benchmarking Hub data19.4%
49Llama 4 Scout· Epoch AI Benchmarking Hub data16%
50DeepSeek V3.1· Epoch AI Benchmarking Hub data5%

Data credits: Epoch AI Benchmarking Hub, CC BY 4.0 (licence)