Benchmark
tau3-bench v1 banking-knowledge pass^2
Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Higher is better.
- Models measured
- 25
- Scores recorded
- 26
- Sources
- 1
- IQ weight
- 1.40
Score distribution
- 22.3%
- Mean
- 23.9%
- Median
- 12%
- Std dev
- 7.2%
- 10th percentile
- 39.2%
- 90th percentile
- 6%
- Min
This benchmark has a weight of 1.40 for chat models, measured over 25 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.
Leaderboard
| # | Model | Score |
|---|---|---|
| 1 | Qwen3.8 Maxsubmission-qwen3-8-max_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 45.2% |
| 2 | Grok 4.5submission-grok-4-5_sierra_2026-08-04, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 39.5% |
| 3 | Claude Opus 5submission-claude-opus-5_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 39.2% |
| 4 | GPT 5.5submission-gpt-5-5_sierra_2026-05-05, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 37.6% |
| 5 | GPT-5.6 Solsubmission-gpt-5-6-sol_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 37.3% |
| 6 | Claude Fable 5submission-claude-fable-5_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 32.8% |
| 7 | Claude Opus 4.8submission-claude-opus-4-8_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 31.3% |
| 8 | Claude Opus 4.7submission-claude-opus-4-7_sierra_2026-05-05, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 31.1% |
| 9 | Muse Spark 1.1submission-muse-spark-1-1_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 29.6% |
| 10 | GPT 5.4submission-gpt-5-4_sierra_2026-03-25, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 29.6% |
| 11 | GLM 5.2submission-glm-5-2_sierra_2026-08-04, usersim-gpt-5.2, effort-xhigh, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 25.3% |
| 12 | Kimi K3submission-kimi-k3_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 25.3% |
| 13 | GPT 5.2submission-gpt-5-2_sierra_2026-02-26, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 23.9% |
| 14 | Claude Opus 4.6submission-claude-opus-4-6_sierra_2026-05-05, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 18.7% |
| 15 | Claude Opus 4.5submission-claude-opus-4-5_sierra_2026-02-26, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 17.2% |
| 16 | Inkling FP4submission-inkling_sierra_2026-08-04, usersim-gpt-5.2, effort-max, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 16.7% |
| 17 | Claude Sonnet 4.5submission-claude-sonnet-4-5_sierra_2026-02-26, usersim-gpt-5.2, effort-enabled, retrieval-terminal· Sierra tau-bench Leaderboard data docs | 15.8% |
| 18 | Gemini 3 Flashsubmission-gemini-3-flash_sierra_2026-03-02, usersim-gpt-5.2, effort-high, retrieval-terminal· Sierra tau-bench Leaderboard data docs | 15.5% |
| 19 | Gemini 3.1 Pro Previewsubmission-gemini-3-1-pro-preview_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 15.1% |
| 20 | Gemini 3 Pro Previewsubmission-gemini-3-pro_sierra_2026-03-02, usersim-gpt-5.2, effort-high, retrieval-terminal· Sierra tau-bench Leaderboard data docs | 9.3% |
| 21 | Grok 4 Fastsubmission-grok-4-fast_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 8.4% |
| 22 | Grok 4.1 Fastsubmission-grok-4-1-fast_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 7.7% |
| 23 | GLM 5submission-glm-5-think_sierra_2026-03-02, usersim-gpt-5.2, effort-enabled, retrieval-text-emb-3-large· Sierra tau-bench Leaderboard data docs | 7.2% |
| 24 | GPT 5.2submission-gpt-5-2-none_sierra_2026-02-26, usersim-gpt-5.2, effort-none, retrieval-qwen-embeddings· Sierra tau-bench Leaderboard data docs | 7.2% |
| 25 | Qwen3.5 397B A17Bsubmission-qwen3.5-397b-a17b-think_sierra_2026-03-02, usersim-gpt-5.2, effort-enabled, retrieval-text-emb-3-large· Sierra tau-bench Leaderboard data docs | 6.4% |
| 26 | Gemini 2.5 Prosubmission-gemini-2-5-pro_sierra_2026-05-05, usersim-gpt-5.2, effort-high, retrieval-alltools· Sierra tau-bench Leaderboard data docs | 6% |
Data credits: Sierra Research tau2-bench leaderboard submissions (github.com/sierra-research/tau2-bench), MIT (licence)