BPCO Trains an LLM Critic With a Single Rollout, Replacing GRPO’s Within-Group Comparison Across Multiple Responses

A research team at the National University of Singapore has introduced Best-Practice Critic Optimization (BPCO), revisiting why recent LLM reinforcement learning has shifted toward critic-free methods such as GRPO. GRPO generates multiple responses for the same prompt and estimates advantages from their relative rewards within the group. This eliminates the value model, but requires multiple rollouts per prompt and typically applies the same outcome signal across the entire response. BPCO instead trains a critic to estimate the expected return of each response prefix, allowing it to construct per-token signals from just one sampled response.

The [paper](https://arxiv.org/abs/2608.23566) does not introduce a single new loss function. Instead, it assembles several design choices that can easily become misaligned into a complete recipe: using DPPO to constrain absolute changes in the probabilities of sampled tokens; applying a scaled arctangent to restrict value predictions to the known reward range; using the final observed outcome as an unbiased Monte Carlo target for the critic; removing batch advantage normalization to avoid amplifying tiny amounts of noise as training approaches convergence; and adapting GAE to response length so terminal rewards do not decay differently across shorter and longer sequences.

Because the critic is used only during training, it can also access privileged inputs unavailable to the policy, such as reference answers, solution steps, or grading rubrics. These inputs can improve value estimation without changing the inputs required at deployment.

The authors evaluated BPCO on the DeepScaleR 40.3K dataset, DAPO-Math-17K, and OpenRubrics, using the DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-30B-A3B model families. The results show that BPCO consistently outperforms their critic baseline and, with one response per prompt, matches or exceeds group baselines that sample multiple responses. The team has also released [training code and execution scripts](https://github.com/QPHutu/golden_critic) based on verl, covering mathematics, MoE, and rubric experiments.

However, “single rollout” only means generating fewer trajectories. The critic still adds parameters, forward passes, activations, optimizer state, and memory usage, and the paper does not provide comprehensive comparisons of wall-clock time, GPU hours, or peak memory. The evidence is also concentrated on mathematics and rubric tasks with known reward ranges, while privileged inputs may accelerate overfitting in low-data settings. The most important next step is to test whether this recipe remains more stable than GRPO—and genuinely reduces total costs—on agentic tool-use and coding tasks, as well as under noisy or unbounded rewards.