Back Home

AI 安全

Microsoft’s Lazarus Research Shows How Modifying a Few Neurons Can Implant an Agent Backdoor

On November 10, Microsoft disclosed Lazarus, demonstrating how directly modifying a small number of model neurons can make an agent output malicious instructions when it encounters a specified string. The research identifies model weight integrity as a deployment risk, while noting clear limits on experimental success rates and detection capabilities.

miss_rogue on flickr, https://www.flickr.com/photos/missrogue/132777293/ · CC BY-SA 2.0 · Image source
zh-Hant

On November 10, Microsoft researchers disclosed Lazarus, demonstrating that an attacker with write access to a model checkpoint can modify a small number of dormant neurons so that an otherwise normally functioning agent outputs attack instructions when it encounters a specified string. If the agent can execute terminal commands or use tools, that output could lead to real-world actions. Microsoft said it has not observed threat actors weaponizing this technique to date. Research announcement, security guidance

The method targets neurons that are usually inactive in the model’s final feed-forward block, repurposing them as trigger detectors. Once activated, they use the residual stream to increase the likelihood of specific output tokens. The researchers say a payload containing P tokens requires approximately P neurons. The modifications can be calculated directly, or produced by running hundreds of steps of gradient adjustment on local weights. So “no model retraining required” does not mean every experiment involved no optimization at all. Paper abstract

In experiments with Nemotron-Terminal-8B, the authors used 23 trajectories, both with and without triggers, to modify seven of more than 12,000 neurons. On held-out test trajectories, the payload trigger rate was at least 98.8%; benign trajectories did not trigger. These results apply to the specified model and test distribution and should not be generalized to claim that all agents would be vulnerable at the same rate. The authors also released a preprint.

Defenses involve trade-offs as well: without access to the original reference model, the agent defense in the study flagged 91.4% of held-out malicious checkpoints, with a 9.7% false positive rate. For deployment teams, this supports adding weight audits to the model intake process, but the results are not enough to treat a scan as a security guarantee. Engineering steps include keeping a verifiable base version, comparing weights before and after updates, and restricting agents’ tool permissions and network egress. Follow-up questions include whether the findings can be independently reproduced and how detection methods can distinguish legitimate fine-tuning from anomalous modifications. Experimental summary, deployment guidance

Sources

  1. Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents
  2. Lazarus 預印本
  3. Lazarus 論文摘要與實驗結果
  4. Detecting sleeper agents hiding in AI model neurons