代理安全／強化學習評估
HERALD Reveals That Search Agents Can Cite Unretrieved Passages; A Single Membership Check Reduces the Observed Attack Rate to Zero
HERALD audits search-agent rewards using question-matched counterfactual edits, finding that citations to real but never-retrieved passages can score at least as highly as the original trajectory under the baseline reward. After adding a citation–retrieval-set membership check, no successful attacks were observed across 593 eligible questions, but the reinforcement signal appeared in only 0.03% of training trajectories.
Search-agent rewards typically combine answer correctness, citation support, tool cost, formatting, and anti-cheating criteria. HERALD’s central finding is that even an apparently comprehensive aggregate score may still permit “citation laundering”: the answer and search history remain unchanged, while citations are replaced with passages that genuinely exist in the corpus but were never retrieved by the agent. Because other rewards—or penalties removed by the substitution—can offset the violation, the score does not necessarily decrease.
The researchers treated the reward function as a program that can accept paired unit tests and constructed three question-matched interventions: removing search, fabricating nonexistent citation IDs, and substituting real but unretrieved IDs. Across four pools of Qwen3-8B trajectories from HotpotQA, 2WikiMultiHopQA, and MuSiQue, they first excluded samples that already violated the rules or in which an intervention simultaneously changed other conditions, leaving 593 question clusters. The baseline reward could already reject search removal and fake IDs. However, a label-free lexical attack still achieved a 4.30% success rate, rising to 6.66% when an offline oracle selected among eight candidates. When the BM25 candidate pool was expanded to 16, the oracle upper bound reached 13.74%.
After exhaustively enumerating all eight combinations of three detectors, the minimal fix was not to add more rules but to strengthen `L`: checking whether a citation ID actually appeared in the previously retrieved set. After this change, no successful attacks were observed, with a one-sided 95% upper bound of 0.50%. By contrast, the full rule set, which included additional oracle-annotated penalties, still had a 3.37% attack rate because replacing citations could simultaneously remove existing penalties. This shows that adding more rules does not guarantee monotonically improving safety.
In paired Search-GRPO experiments using approximately five million generated tokens per group, the new reward increased mean citation precision by 2.02 points and support recall by 1.46 points across the three datasets, while reducing unsupported citations by 1.69 points. However, MuSiQue did not meet the non-inferiority threshold for answer exact match (EM). More importantly, `L` fired on only 18 of 58,368 training trajectories, and some of that signal was further canceled out by group normalization. This audit can establish only the existence of a scoring vulnerability under a closed corpus with structured IDs; it cannot verify whether citations semantically support an answer, and it does not yet cover live web search, free-form citations, or corpus contamination.