Skip to content

Benchmark

AgentDojo Targeted ASR

Agentic benchmarks measure how well models carry out tasks that take several steps. Scores are percentages. Lower is better.

Models measured
19
Scores recorded
28
Sources
1
IQ weight
0.00

Score distribution

18.8%
Mean
17%
Median
14.6%
Std dev
3.7%
10th percentile
34.5%
90th percentile
1.1%
Min
1.1%56.3%
The band covers the middle 80% of scores. The line marks the median.

This benchmark has a weight of 0.00 for chat models, measured over 19 models. The weight is how much it counts toward llminfo IQ. We measure it from the data, not from the raw scores.

Leaderboard

#ModelScore
1Claude 3.5 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-11-15· AgentDojo results data docs1.1%
2Command R+provider: cohere, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs3.3%
3gemini-1.5-flash-002provider: google, defense: None, attack: important_instructions, date: 2024-11-05· AgentDojo results data docs3.5%
4GPT 4oprovider: openai, defense: None, attack: direct, date: 2024-06-05· AgentDojo results data docs3.7%
5Command R Plusprovider: cohere, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs4.5%
6GPT 4oprovider: openai, defense: None, attack: ignore_previous, date: 2024-06-05· AgentDojo results data docs5.4%
7GPT 4oprovider: openai, defense: None, attack: injecagent, date: 2024-06-05· AgentDojo results data docs5.7%
8GPT 4oprovider: openai, defense: tool_filter, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs6.8%
9Claude Sonnet 3.7provider: anthropic, defense: None, attack: important_instructions, date: 2025-02-24· AgentDojo results data docs7.3%
10GPT 4oprovider: openai, defense: transformers_pi_detector, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs8%
11Claude 3 Haikuprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs9.1%
12GPT-3.5 Turboprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs10.3%
13Claude 3 Opusprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs11.3%
14gemini-1.5-flash-001provider: google, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs12.2%
15Gemini 2.0 Flash Experimentalprovider: google, defense: None, attack: important_instructions, date: 2024-12-11· AgentDojo results data docs17%
16gemini-1.5-pro-002provider: google, defense: None, attack: important_instructions, date: 2024-11-04· AgentDojo results data docs17%
17Gemini 2.0 Flashprovider: google, defense: None, attack: important_instructions, date: 2025-01-30· AgentDojo results data docs20.8%
18Llama-3-70b-chat-hfprovider: together-prompting, defense: None, attack: important_instructions, date: 2021-09-05· AgentDojo results data docs25.6%
19Claude 3 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs26.7%
20GPT-4o miniprovider: openai, defense: None, attack: important_instructions, date: 2024-07-19· AgentDojo results data docs27.2%
21GPT 4oprovider: openai, defense: repeat_user_prompt, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs27.8%
22gemini-1.5-pro-001provider: google, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs28.6%
23GPT-4 Turboprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs28.6%
24Claude 3.5 Sonnetprovider: anthropic, defense: None, attack: important_instructions, date: 2024-06-20· AgentDojo results data docs33.9%
25GPT 4oprovider: openai, defense: None, attack: tool_knowledge, date: 2024-06-05· AgentDojo results data docs34.5%
26GPT 4oprovider: openai, defense: spotlighting_with_delimiting, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs41.7%
27GPT 4oprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs47.7%
28GPT-4-0125-previewprovider: openai, defense: None, attack: important_instructions, date: 2024-06-05· AgentDojo results data docs56.3%

Data credits: AgentDojo. Copyright (c) 2024 Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. MIT License. Not a leaderboard: attack and defense coverage differs between models. (licence)