代理評測
Cross-Language Prompts Alter Agent Tool Paths, With Four Large Models Retaining Only 71%–73% of Their Action Policies
A Microsoft research team compared 41 languages across 2.38 million rollouts and found that agents take different tool paths when the same task is presented in another language—even with greedy decoding. The study also shows that non-English tasks rely heavily on first translating into English, although its tools are merely symbolic calls and do not capture the errors or side effects of real-world APIs.
Multilingual agent evaluations typically compare only final answers, overlooking the intermediate actions that actually determine latency, cost, and permission risks. A new measurement pipeline proposed by a Microsoft research team defines a shared symbolic action space comprising five tool names. Using six parallel benchmarks, 41 languages, and eight models, the team generated 2,382,875 rollouts, then compared trajectories from repeated runs of the same task in the same language with those produced across languages.
Directly calculating trajectory similarity is easily distorted by short trajectories, empty trajectories, chance overlap, and the model’s own instability. The team therefore excluded empty trajectories, matched trajectory lengths, and divided cross-language consistency by same-language reproducibility to obtain a “normalized policy retention rate.” At temperature zero, Gemma 3 27B, Sarvam-M, Qwen3-235B-A22B, and Llama 4 Maverick fell within a narrow range of 71%–73%. Model identity explained only 5.7% of the variance, while benchmark differences explained 26.9%. Models with fewer than 10B parameters did not exhibit the same pattern, and failing to correct for chance overlap could even reverse the rankings of smaller models.
The trajectories show that non-English inputs generally pass through English as a pivot language: Translate was the most frequently used tool in the adapted benchmarks, and roughly 99% of the reasoning text was ASCII. Removing the translation tool reduced cross-language consistency in proportion to how heavily each model had originally used it, while compliance with direct instructions not to translate remained below 1%. Another warning sign came from the evaluation pipeline: a regular expression unsuitable for multilingual output had produced false failures. After two examples were added, one model’s parseable accuracy increased 26-fold, even though its human-readable output changed little.
From an engineering perspective, tool policies, cost estimates, and safety tests validated in English cannot be directly extrapolated to other languages. Systems should also record action sequences, within-language reproducibility, and parsing failures. One limitation is that none of the tool calls were actually executed, so the study did not measure API responses, retries, state changes, or permission-related consequences. The paper states that the code will be released publicly, but the GitHub repository is currently empty; only the Hugging Face benchmark data can be inspected directly.