DUMA-Bench

Properties

axis: multi-agent
verification: mixed
whatItTests: Agent security benchmark for dual-control interaction: extends tau2-bench with adversarial environments covering eight vulnerability classes (including RAG poisoning, cross-agent manipulation, and unsafe output handling), evaluating 14 models from five model families across eight domains and multiple user-behavior regimes, where both the agent and a simulated user can influence shared environment state.
saturation: open
patternLabFit: Its central finding, that dual-control interaction raises attack success rate from 26.9% to 41.1%, is direct evidence that a collaboration pattern's security surface is not just a property of the model but emerges from agent-environment-user interaction; relevant to any Pattern Lab pattern where an agent shares write access to state with another party.
source: DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security (arXiv:2609.24662)
notes: Submitted 2026-09-21, within the discovery window. Distinct from every tracked multi-agent entry (Sotopia, MAgIC, COMMA, LLM-Coordination, Alem, GPTNT, MultiAgentBench, BattleAgentBench, Bazaar): DUMA-Bench targets agent security specifically, with an explicit taxonomy of eight vulnerability classes under dual-control (agent-plus-user) state manipulation, a mechanism absent from the cooperative or competitive-task benchmarks already tracked. Verification marked mixed rather than det-test: the abstract reports an 'attack success rate' but does not state whether success is adjudicated by a deterministic check or a judge, and security benchmarks of this kind commonly use a judge to assess compliance. Builds on tau2-bench (arXiv:2506.07982), which is not itself proposed as a tracked entity in this submission.

Provenance

Source: https://arxiv.org/abs/2609.24662

Properties

axis

multi-agent

verification

mixed

whatItTests

Agent security benchmark for dual-control interaction: extends tau2-bench with adversarial environments covering eight vulnerability classes (including RAG poisoning, cross-agent manipulation, and unsafe output handling), evaluating 14 models from five model families across eight domains and multiple user-behavior regimes, where both the agent and a simulated user can influence shared environment state.

saturation

open

patternLabFit

Its central finding, that dual-control interaction raises attack success rate from 26.9% to 41.1%, is direct evidence that a collaboration pattern's security surface is not just a property of the model but emerges from agent-environment-user interaction; relevant to any Pattern Lab pattern where an agent shares write access to state with another party.

source

DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security (arXiv:2609.24662)

notes

Submitted 2026-09-21, within the discovery window. Distinct from every tracked multi-agent entry (Sotopia, MAgIC, COMMA, LLM-Coordination, Alem, GPTNT, MultiAgentBench, BattleAgentBench, Bazaar): DUMA-Bench targets agent security specifically, with an explicit taxonomy of eight vulnerability classes under dual-control (agent-plus-user) state manipulation, a mechanism absent from the cooperative or competitive-task benchmarks already tracked. Verification marked mixed rather than det-test: the abstract reports an 'attack success rate' but does not state whether success is adjudicated by a deterministic check or a judge, and security benchmarks of this kind commonly use a judge to assess compliance. Builds on tau2-bench (arXiv:2506.07982), which is not itself proposed as a tracked entity in this submission.