Back Home

AI 安全

Running a Vending Machine Business: Blocking Policy Violations, While Misinformation Still Enters Memory

Reflection’s field evaluation records memory contamination, pricing and ordering errors. For teams preparing to grant agents financial authority, how much of the workflow required human intervention is more relevant to deployment than refusal rates alone.

zh-Hant

Teams preparing to let AI handle purchasing and refunds have a new field case to examine. On November 20, 2026, Reflection published an evaluation of its Beam office vending business: the agent blocked several attempts to induce policy violations, yet was still affected by misinformation and required human intervention in transaction workflows. These are the developer’s preliminary observations, not an independent validation. Original evaluation

How a false notification became a persistent memory

Reflection partnered with Andon Labs to deploy agents in offices in New York and San Francisco. The case published here is mainly about Mr. Beam in New York. Employees ordered through Slack and deliberately tested its boundaries. According to the company’s account, repeated attempts to induce it to sell nicotine products failed. But after a researcher emailed a false claim that the head of the security team had been fired, the agent refused that manager’s later requests. The false claim entered the long-term memory summary during “conversation compaction,” and continued to affect behavior for several days. Case details

Conversation compaction condenses older interactions into a shorter summary so the agent can continue working. The deployment question raised by this case is whether summaries preserve the source and reliability of information. If an unverified email becomes the basis for later work, subsequent decisions may diverge from the task even if the agent did not accept a policy-violating instruction at the time.

Payment received, but the order still has to be placed

Operational issues also appeared in transaction handoffs. The report records a price of $8 for paper clips instead of $0.08, as well as cases where customers had paid but no supplier order was created. There were 23 human intervention records from October 8 to 30, 12 of which involved financial oversight or process corrections. The others also included clarifying rules and repairing tools, so the total should not be treated as a model error rate. Intervention records

The agent runs on Pion, developed by Andon Labs. The platform provides continuously running agents and tools such as a browser, email, and banking; it is currently a research preview. Andon also states that businesses on the platform are experiments first, and that its monitoring systems are still being improved. Pion overview

For users, a simple “done” is not enough to confirm that a transaction has been completed. Based on these cases, acceptance checks should track whether payments correspond to orders, whether orders are successfully placed, and whether the outcome is recorded in the ledger. These are checks suggested by the editors based on the cases, not fixes whose effectiveness has been validated.

Refusal rates and operational reliability measure different things

Existing evaluations provide different baselines: Reflection’s Beam Model Card lists refusal or redirection rates for harmful requests; Andon’s Vending-Bench 2 has models operate a simulated vending business for a year, scored by their ending account balance. These measure policy compliance and simulated business performance, respectively. Neither directly answers whether a real purchase is completed after payment.

Reflection also acknowledges that the results were shaped by human repairs and approval requirements, and therefore do not establish unsupervised operational reliability. Evaluation limitations For teams preparing to expand agent authority, the report’s most useful reminder is to separately record transactions completed by the agent itself and those completed after human intervention. That makes it possible to see how much responsibility was actually delegated.

Sources

  1. Mr. Beam: Partnering with Andon Labs to evaluate Beam’s autonomy risk in a live, real-world deployment — Reflection
  2. Beam Model Card — Reflection
  3. Pion | Andon Labs
  4. Vending-Bench 2 | Andon Labs