PILOT Manages Taobao Recommendation Experiments with Deterministic Guardrails, Raising Search Efficiency from 53.3% to 93.3%

The Taobao team introduced PILOT, a system that lets LLM agents participate in experiment design for online recommendation systems rather than directly generating recommendation lists. The system has three roles: the Experiment Manager handles requirement confirmation, monitoring, incident response, and experiment closure; the Search Planner uses decision trees to propose candidate strategies for different user segments; and the Memory Curator, once results are finalized, organizes strategy differences and experimental methods into memories annotated with sources and confidence levels.

The key architectural choice is not to give agents more authority, but to constrain their decision space. Manager Guard uses a state machine and freeze policies to generate a set of valid instructions, from which the LLM can only select. State Committer is the only component permitted to write state. Statistics Engine generates decision certificates that agents cannot modify, while Contract Builder locks in the conditions for success, failure, or an extended observation period before data collection begins. Candidate Validator and Action Enumerator prevent the Planner from inventing unauthorized actions. High-risk fields—including estimators, traffic budgets, stopping rules, and final rollout decisions—still require human confirmation.

The search process uses a fixed baseline, B0, together with a Champion–Challenger structure. Each Challenger may contain only one atomic change relative to the Champion and receives a promote, reject, or continue certificate under preregistered rules. Metric direction is withheld from decision-making until the observation window has actually ended, reducing the risks of peeking at results, cherry-picking dates, or repeatedly changing thresholds. Memory also uses task branches and a merge mechanism, allowing only validated content to enter the shared knowledge base.

The team compared PILOT with the free-form exploration system ROAM across five experiment buckets in the “Guess What You Like” section of Taobao’s home page. The report says search efficiency rose from 53.3% to 93.3%. In the best-performing bucket, product detail page views, core views, transaction count, and gross merchandise value increased by 1.40%, 1.60%, 0.96%, and 1.50%, respectively. The technical significance lies in demonstrating how LLM-based semantic judgment can be embedded in existing A/B testing infrastructure while keeping irreversible operations within auditable services.

However, these are self-reported online results from a single platform. The paper does not disclose traffic volumes, confidence intervals, model costs, or the complete codebase, and five buckets are insufficient to demonstrate generalization across scenarios. Engineering teams should next examine guardrail misclassifications, agent latency, memory contamination, and whether the gains are large enough to offset the additional inference and experimental traffic costs.