AI 安全／代理系統
SafeKeep Finds Tool Schemas Weaken Models’ Refusal Signals, Cutting Prompt Injection Success to 2.5%
New research suggests that the JSON-style tool specifications used by agents are not merely neutral interfaces and may suppress models’ internal refusal representations. By separately evaluating safety before execution using flattened text specifications, SafeKeep increased the average refusal rate for harmful requests from 23.8% to 70.6%.
Tool calling is usually treated as an engineering layer that converts model output into structured parameters, but the SafeKeep team found that simply expressing tool capabilities in schema format can alter a model’s safety behavior. Using white-box hidden-state analysis, the researchers compared ordinary chat prompts with agent prompts and observed that adding tool specifications weakened the models’ existing internal refusal direction. After replacing the schema’s semantics with meaningless text or decomposing the agent prompt into separate components, they found that the structure itself remained an important factor in the decline in safety.
SafeKeep does not modify the actual tool interface. Instead, it separates safety evaluation and execution into two paths: a classifier first reads tool descriptions flattened from JSON into natural language and determines whether the request should be refused; if the request passes, the execution model still uses the original schema to generate a valid tool call. This avoids breaking parameter constraints for the sake of safety checks and allows SafeKeep to be placed at the inference front end of existing agent frameworks.
According to the team’s reported results across two benchmarks and four white-box and black-box models, the average refusal rate for harmful requests rose from 23.8% to 70.6%, while the average attack success rate for observed content-level prompt injections fell from 25.6% to 2.5%. The public code includes 400 paired requests, representation-direction extraction, causal interventions, and an end-to-end demonstration. It also pins the Transformers version because the way chat templates render tools is itself one of the variables under study.
The findings are a reminder to agent developers that safety regression testing must cover the actual tool schemas, template versions, and serialization formats rather than testing only the bare model. However, the current repository is small, and the primary white-box reproductions use Llama 3.1 8B Instruct and Qwen2.5 7B Instruct. Whether the higher refusal rate will lead to false positives, additional latency, or classifier bypasses in large production toolsets still requires independent validation.