模型訓練與最佳化
Hyper-ES First Uses Gradients to Build a Low-Dimensional Search Space, Averaging 1% Higher on Mathematical Reasoning Than GRPO-LoRA
Rather than applying random evolution directly to billions of model-weight dimensions, Hyper-ES constructs a subspace from a small number of fine-tuning directions and then searches for layer-wise model-merging coefficients. The authors report modest gains across three models and six mathematical reasoning datasets, although the method is not entirely gradient-free.
The newly proposed Hyper-ES attempts to address the difficulty of scaling evolution strategies directly to large language models. A traditional Evolution Strategy estimates update directions using random perturbations. When the parameter space reaches billions of dimensions, most perturbations are nearly orthogonal to genuinely useful descent directions, causing sample efficiency and stability to deteriorate rapidly. Instead of blindly searching the full weight space, Hyper-ES first performs a small number of relatively inexpensive gradient-based fine-tuning runs to obtain multiple descent directions that individually offer limited gains but contain task-specific information.
The linear span of these directions becomes the search space for the subsequent evolutionary process. The system uses CMA-ES to optimize layer-wise coefficients, then combines multiple update directions into the final model through DARE-TIES-style model merging. In other words, the evolution strategy searches for “how to mix directions already known to be useful,” rather than rediscovering how billions of parameters should move. This makes the dimensionality of the black-box optimization depend on the number of candidate directions and layers, which in theory is better suited to resource-constrained post-training than full-parameter perturbation.
The authors evaluated three backbone models from the Qwen2.5-Instruct and DeepSeek-R1-Distill families across six mathematical reasoning datasets. They report that Hyper-ES scores about 1% higher on average than GRPO-LoRA while using 10% fewer gradient updates that require saving intermediate states. The primary technical value lies not in the single percentage point itself, but in connecting model merging, low-dimensional subspaces, and evolution strategies into a practical post-training pipeline. The approach may also be useful when rewards are non-differentiable, evaluators are complex, or reducing the number of backward passes is desirable.
However, Hyper-ES still requires gradient-based fine-tuning to generate its basis, so it should not be considered a purely black-box or zero-backpropagation method. The current results also cover only mathematical reasoning and three models, and do not yet demonstrate comparable benefits for coding agents, open-ended generation, or larger models. Future work should compare total GPU hours, peak memory usage, and search-evaluation costs, while testing whether the method truly retains its low-dimensional advantage as the number of directions increases.