Skip to content

Benchmark

tau3-bench v1 banking-knowledge pass^2

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.

Models measured
25
Scores recorded
26
Sources
1
IQ weight
1.40

Score distribution

22.3%
Mean
23.9%
Median
12%
Std dev
7.2%
10th percentile
39.2%
90th percentile
6%
Min
6%45.2%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 1.40 for chat models, measured over 25 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Qwen3.8 Maxsubmission-qwen3-8-max_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs45.2%
2Grok 4.5submission-grok-4-5_sierra_2026-08-04, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs39.5%
3Claude Opus 5submission-claude-opus-5_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs39.2%
4GPT 5.5submission-gpt-5-5_sierra_2026-05-05, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs37.6%
5GPT-5.6 Solsubmission-gpt-5-6-sol_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs37.3%
6Claude Fable 5submission-claude-fable-5_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs32.8%
7Claude Opus 4.8submission-claude-opus-4-8_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs31.3%
8Claude Opus 4.7submission-claude-opus-4-7_sierra_2026-05-05, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs31.1%
9Muse Spark 1.1submission-muse-spark-1-1_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs29.6%
10GPT 5.4submission-gpt-5-4_sierra_2026-03-25, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs29.6%
11GLM 5.2submission-glm-5-2_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs25.3%
12Kimi K3submission-kimi-k3_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs25.3%
13GPT 5.2submission-gpt-5-2_sierra_2026-02-26, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs23.9%
14Claude Opus 4.6submission-claude-opus-4-6_sierra_2026-05-05, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs18.7%
15Claude Opus 4.5submission-claude-opus-4-5_sierra_2026-02-26, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs17.2%
16Inkling FP4submission-inkling_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs16.7%
17Claude Sonnet 4.5submission-claude-sonnet-4-5_sierra_2026-02-26, usersim-gpt-5.2, effort-enabled, retrieval-terminal· Sierra tau-bench Leaderboard data docs15.8%
18Gemini 3 Flashsubmission-gemini-3-flash_sierra_2026-03-02, usersim-gpt-5.2, effort-high, retrieval-terminal· Sierra tau-bench Leaderboard data docs15.5%
19Gemini 3.1 Pro Previewsubmission-gemini-3-1-pro-preview_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs15.1%
20Gemini 3 Pro Previewsubmission-gemini-3-pro_sierra_2026-03-02, usersim-gpt-5.2, effort-high, retrieval-terminal· Sierra tau-bench Leaderboard data docs9.3%
21Grok 4 Fastsubmission-grok-4-fast_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs8.4%
22Grok 4.1 Fastsubmission-grok-4-1-fast_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs7.7%
23GLM 5submission-glm-5-think_sierra_2026-03-02, usersim-gpt-5.2, effort-enabled, retrieval-text-emb-3-large· Sierra tau-bench Leaderboard data docs7.2%
24GPT 5.2submission-gpt-5-2-none_sierra_2026-02-26, usersim-gpt-5.2, effort-none, retrieval-qwen-embeddings· Sierra tau-bench Leaderboard data docs7.2%
25Qwen3.5 397B A17Bsubmission-qwen3.5-397b-a17b-think_sierra_2026-03-02, usersim-gpt-5.2, effort-enabled, retrieval-text-emb-3-large· Sierra tau-bench Leaderboard data docs6.4%
26Gemini 2.5 Prosubmission-gemini-2-5-pro_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs6%

Data credits: Sierra Research tau2-bench leaderboard submissions (github.com/sierra-research/tau2-bench), MIT (licence)