模型訓練與對齊
RRC Converts Generative Reward Model Rankings into GRPO Signals, Raising AlpacaEval 2 Score from 35.8% to 41.3%
Instead of treating the probability of a preference token as the reward, RRC constructs scores from the relative rankings of candidate responses. With an 8B reward model and eight-vote aggregation, the ArenaHardV2 score also rises from 8.0% to 11.2%, at the cost of additional reward model inference.
Generative reward models can reason before comparing two responses, yet RL pipelines often compress their judgments into the probability of a single preference token. The newly proposed Ranking-based Reward Construction (RRC) argues that this forces the model’s comparison capabilities into an unsuitable absolute-score interface: chain-of-thought reasoning often causes the final A/B probabilities to saturate, while confidence does not necessarily correspond to response quality. In the study, 88.3% of samples evaluated by the pointwise generative reward model fell within either 0–0.1 or 0.9–1.0, with only 11.7% in the intermediate range.
RRC retains the scalar output required by GRPO but changes how it is derived. Self-Competitive Ranking compares multiple rollouts for the same prompt against one another and assigns rewards based on the number of wins. If the comparisons form a cycle—A beats B, B beats C, and C beats A—the method uses a weighted graph and approximate Kemeny aggregation to produce a consistent ordering. Anchor-Guided Ranking instead generates a small number of anchors using a fixed reference policy. Each candidate then needs only to be evaluated by how many anchors it beats, avoiding exhaustive pairwise comparisons. Majority voting over multiple samples can also trade additional inference budget for more stable preference judgments.
The authors used Llama 3.1 8B as the policy model, 3B and 8B generative reward models, and 7,500 RL examples. With the 8B reward model, the probability-based baseline scored 35.8%, 8.0%, and 52.9% on AlpacaEval 2, ArenaHardV2, and MMLU-Redux, respectively. Anchor-Guided Ranking with voting@8 raised those scores to 41.3%, 11.2%, and 56.9%. Probability aggregation using approximately the same amount of compute still performed worse, supporting the conclusion that the gains do not come solely from additional inference passes.
The public implementation integrates with GRPO through an MS-SWIFT plugin, serves the reward model through an OpenAI-compatible API, and includes an example of multi-port deployment with vLLM. Engineering teams must still account for the cost: pairwise comparisons, voting, and anchors all increase the number of inference calls. The repository currently contains only initial commits, does not include the reward model weights used in the authors’ experiments, and still has a placeholder string in the README’s clone command. Future work should track independent reproductions, larger rollout groups, and whether ranking-model biases are amplified by the policy during long-term training.