生成模型訓練
DiffusionOPSD Turns Image Rewards Into Stepwise Denoising Targets, Cutting Training GPU Hours by Up to 63%
ByteDance Seed’s method avoids applying a terminal reward directly to every denoising step. Instead, it uses reward gradients to construct positive and negative clean-image prediction targets. It achieved the best result in 19 of 20 paired tests across two backbones, although the findings still rely primarily on automated reward models and measurements conducted by the authors.
Preference post-training for diffusion models typically receives a reward only after a complete image has been generated, yet must attribute that same signal across the entire denoising trajectory. [DiffusionOPSD](https://arxiv.org/abs/2608.24646), proposed by ByteDance Seed, instead uses on-policy self-distillation. In each round, it first freezes the behavior policy to generate trajectories, then samples a noisy query state and a clean-output prediction as an anchor. It follows the image-reward gradient to construct bounded positive and negative targets around the anchor. The trainable policy treats these targets as stop-gradient supervision, and after a finite number of fitting steps, the behavior policy is updated using an exponential moving average.
The technical value of this layered design goes beyond its scores. Researchers can separately measure whether the reward gradient identifies a better target and whether the model can realize that improvement after an actual update. The paper’s same-query experiments show that targets with larger improvements during construction do not necessarily produce larger gains after a finite number of parameter updates, indicating that target quality and learnability are distinct problems in diffusion RL.
The authors conducted paired reward comparisons on SD3.5-M and the native nine-step Z-Image-Turbo using ten evaluators. DiffusionOPSD achieved the best final score in 19 of 20 held-out comparisons, with improvements of up to 44% over the strongest baseline. Re-estimated from per-step timings on eight GPUs, every 100 updates required 28.2 and 149.8 GPU-hours, respectively—40% and 63% less than DiffusionNFT. However, peak GPU memory usage for Z-Image-Turbo did not decrease accordingly.
The [public code](https://github.com/worldbench/DiffusionOPSD) includes training configurations for SD3.5-M, Z-Image-Turbo, ReFL, FlowGRPO, and DiffusionNFT, along with a fixed manifest of 25,415 Pick-a-Pic prompts. It also provides three LoRA checkpoints for rerunning the DrawBench held-out evaluation. The main limitation is that the evidence still comes primarily from two backbones, text-to-image generation, and automated preference evaluators, making it susceptible to reward over-optimization. The paper’s evaluator corresponding to one pointwise checkpoint is also absent from the repository. The next things to watch are independent reproductions, human preference evaluations, and whether the method can extend to video or image-editing models.