DUMA-Bench Adds User Interaction Tests, but Paper and Repository Scores Still Need Alignment

Researchers from ITMO University’s AI Security Lab and other institutions released DUMA-Bench on September 21, extending agent safety evaluation to scenarios in which users and agents jointly change the environment. The paper covers eight safety domains, 35 executable scenarios, and 14 models, focusing on safety rule violations during multi-turn interactions. [Paper](https://arxiv.org/abs/2609.24662)

Built on τ²-bench, the project provides a solo execution mode without an active user and a dual-control mode that includes a user simulator. Each domain includes policies, tools, tasks, and mutable state. Attack content can be hidden in retrieved documents, collaborator messages, or tool interfaces. Users can therefore influence subsequent actions throughout the workflow, beyond simply providing an opening prompt. [Public repository](https://github.com/ai-security-lab-itmo/duma-benchmark)

Scoring examines both the environment and the conversation. Code checks whether execution leaves behind states such as unauthorized refunds, while text responses can be evaluated by an LLM judge for sensitive information disclosure. This allows cases in which an agent refuses an action but leaks information in its explanation to count as failures. The repository also provides a reliability metric for repeated trials, requiring the same task to pass across multiple runs. [Scoring design](https://github.com/ai-security-lab-itmo/duma-benchmark)

The paper reports that enabling the full interaction mechanism increased the aggregate attack success rate from 26.9% to 41.1%, a rise of 14.2 percentage points. However, the corresponding table in the README examined for this article lists 25.4% and 40.3%. The two sources show the same direction of change, but their figures differ. Anyone reproducing the results must check the task sets, evaluators, and result versions rather than treating them as a single, reconciled dataset. [Paper results](https://arxiv.org/html/2609.24662v1), [Repository results](https://github.com/ai-security-lab-itmo/duma-benchmark)

The study also acknowledges that switching modes simultaneously changes multi-turn dialogue, user participation, and tool-induced state changes, making it impossible to isolate each factor’s causal contribution. Users are simulated by models, some judgments rely on LLM judges, and the scenarios are limited in scale. These figures therefore describe only the tested setup and cannot be directly interpreted as the probability of a real product being attacked. [Study limitations](https://arxiv.org/html/2609.24662v1)

For agent developers, a practical next step is to apply the same policies in both static and interactive tests, retain tool side effects and complete conversations, and then examine how results change after adding validation checkpoints. Based on this review, the priority remains reconciling the public results with the executable materials. Fixing model versions, tasks, and judge configurations would make the framework better suited to ongoing safety regression monitoring.