Evolution Strategies Preserve More Reasoning Paths During Post-Training, Outperforming GRPO on Average at Pass@32

Research from institutions including the Southern University of Science and Technology, the National University of Singapore, and Huawei’s Noah’s Ark Lab reevaluates Evolution Strategies (ES) as a method for post-training LLM reasoning. GRPO samples multiple responses from the same policy and constructs token-level gradients based on relative rewards. ES instead adds Gaussian perturbations to the model’s entire parameter set, performs forward-only rollouts, and combines the perturbation directions into an update weighted by standardized rewards. It does not need to retain backpropagation state, requires less memory, and makes it easier to distribute population members across multiple compute nodes.

The authors focus not only on Pass@1, but also on Pass@K—the probability that repeated sampling finds at least one correct answer. The experiments post-trained Qwen2.5-1.5B, Llama-3.2-3B, and Qwen2.5-7B on GSM8K, and trained DeepSeek-R1-Distill-Qwen-1.5B on DeepScaleR. GRPO generally produced larger gains in single-sample accuracy, but under the easier setting it fell below the base model in 15 of 18 Pass@16 and Pass@32 comparisons. ES showed smaller changes in entropy, and its average Pass@1, Pass@16, and Pass@32 exceeded the baseline under both settings.

The difficult-math setting shows the trade-off most clearly: GRPO averaged 52.9/74.7/78.0 across four Pass@1/16/32 measurements, compared with 49.9/75.0/78.9 for ES. When ES and GRPO each received half of the update budget and were applied sequentially, ES→GRPO achieved 52.3/75.8/79.2, improving Pass@32 while retaining most of the single-sample gains. The paper attributes this effect to the diversity of the policy population generated by parameter perturbations and uses verifier-projected Jensen–Shannon diversity to illustrate its relationship with repeated-sampling coverage.

The study also found that ES produced 40.7 to 44.1 times as much overall parameter drift as GRPO, yet the effective gains were concentrated in a small number of larger updates. Weight distance alone therefore cannot determine whether catastrophic forgetting has occurred. For engineering teams, ES may be better suited to scenarios that require candidate-solution diversity, search, or majority voting, but not necessarily to products that serve only a single decoding attempt. The current evidence covers just four models, the largest with 7B parameters, and is limited to mathematics plus a small number of question-answering and coding benchmarks. The repository listed in the paper was still not publicly accessible at the time of verification, leaving computational cost and reproducibility across teams unconfirmed.