SAPO Lets a Single Autoregressive Model Serve as Both Actor and Critic, Speeding Up Each Agent RL Training Iteration by 33.2%

Agent reinforcement learning typically involves a trade-off between two types of cost: PPO can use a critic for per-episode credit assignment, but it must maintain another model nearly as large as the policy; GRPO eliminates the critic, but must generate multiple trajectories for the same prompt. With long-horizon tasks, sparse rewards, or groups in which every outcome is identical, its advantage estimates may collapse to zero. SAPO, released on August 20, attempts to combine the benefits of both approaches in a single autoregressive model.

SAPO reads value estimates at two causal boundaries in each agent turn: it estimates V(s) before generating the action and Q(s,a) after generating the complete action. The researchers reserve two tokens in the vocabulary and map the difference between their logits to a bounded scalar. These tokens are neither decoded nor sent to the environment. The model samples only one trajectory per task, then uses λ-returns, generalized advantage estimation, and episode-level normalization across batches to propagate terminal rewards back to earlier decisions. The policy, V, and Q losses update the shared backbone in a single backward pass.

In tests using Qwen2.5-1.5B and Qwen2.5-7B across three random seeds, SAPO improved by an average of 15.1 and 12.1 percentage points, respectively, over PPO and GRPO. The 7B version achieved an overall success rate of 94.0% on ALFWorld, along with a score of 88.6% and a success rate of 82.4% on WebShop. Under the same settings, PPO recorded 80.4%, 81.4%, and 68.7% on those three metrics. In the Qwen2.5-1.5B ALFWorld test, iteration time fell from 451.2 seconds with PPO to 301.4 seconds, mainly because SAPO generated fewer trajectories and removed the critic inference and update paths.

This design is particularly appealing for long-horizon tool-using agents because it brings value estimation back into the causal language model itself instead of attaching a separate value head or second model. However, the speed measurements cover only one model and one environment, while training used four H200 GPUs and eight A40 GPUs. The paper has not yet released code, making it impossible to verify the claimed memory savings, the stability of the reserved tokens, or policy–value interference at larger model scales. The next things to watch are independent reproductions and whether the method can transfer to environments with more irregular rewards, such as search and code execution.