代理安全
PolicyGuide Constrains Customer Service Agents with Persistent Workflow Graphs, Raising Four-Run All-Pass Rate from 0.42 to 0.62
PolicyGuide does more than block policy-violating actions before tool calls: it tracks incomplete policy steps during every user turn. It improved reliable success rates across three τ²-bench domains, but adds one LLM verification call per turn, and some procedural-compliance metrics were designed by the authors themselves.
A new study introduces PolicyGuide, a system that compiles customer service policies from natural-language documents into traversable workflow graphs and uses an independent verifier to continuously track execution progress. Unlike guards that check parameters only when a write tool—such as one for issuing refunds, changing flights, or modifying accounts—is about to be called, PolicyGuide can identify the first unsatisfied node earlier when an agent skips identity verification, eligibility checks, user consent, or post-action verification.
During the offline phase, policies and tool lists are compiled, repaired, and validated into fixed workflow packages. During the online phase, a verifier LLM is called once before each user turn. Its inputs include the conversation, tool results, the workflow graph, and request state maintained in code. The verifier can organize multiple unresolved requests simultaneously and output the current node, supporting evidence, permitted state-changing tools, and corrective instructions. If the agent prematurely issues an unauthorized write call, the system intercepts it once and generates a correction. It then releases the one-time gate to prevent workflow deadlock.
Using GPT 5.4 as both the agent and verifier, PolicyGuide increased the cross-domain average `Pass^4` score from 0.42 for unguided ReAct to 0.62 across 278 tasks in τ²-bench’s airline, retail, and telecom domains. In telecom, the score rose from 0.193 to 0.614. `Pass^4` requires all four runs of the same task to succeed, so it emphasizes reliability more than single-run success rate. The same airline workflow graph could be applied to Claude Sonnet 4.6 and Gemini 2.5 Pro without rewriting it; Gemini’s overall `Pass^4` increased from 0.48 to 0.68.
The technical value lies in keeping policy state in an inspectable external control layer instead of relying on the agent’s memory. However, the improvement over ReAct in retail was not statistically significant, and the evaluation covered only simulated English-language customer service interactions. The authors’ custom temporal audit also lacked inter-annotator agreement data from a second annotator. Before deployment, teams will still need to measure the latency and cost introduced by per-turn verification, as well as failure modes caused by outdated or incorrectly compiled workflow graphs.