arXiv:2611.25998v1 Announce Type: new
Abstract: Few-step diffusion models are widely deployed for their efficiency. However, reinforcing them with black-box rewards, as is routine for multi-step models, faces two structural challenges. First, a few-step generator is a sampler rather than a score model, so its velocity is not the forward velocity of the generated distribution. Second, it must be anchored to the undistilled teacher, which shares no transitions with it. Existing few-step RL methods handle the latter with a marginal KL to the teacher, but each pays for the former in its reward credit. We propose Shadow Posterior Matching (SPM), which renoises generated samples into shadow inputs and obtains the credit from their shadow posterior. From the KL-regularized reward objective, we derive a regression loss that turns reward values into a reward direction while anchoring the generator’s outputs to the teacher and aligning the shadow inputs with the deployed inputs. Applied to 4-step generators, SPM achieves higher rewards with less reward hacking than prior few-step RL methods.
Trending
- One Of Apple HomeView’s Best Features Will Have You Cooking Like A Pro
- Does the Workspace Do the Work? A Causal Protocol for Verifying Global-Workspace Computation in a Language Model
- 36 Percent Of Households Have Ditched A Streaming Service Subscription Recently: Here’s Why
- [2604.23212] Learning Curves and Prediction Persistence to Under-Regularization of Spectral Algorithms in Large Dimensions
- Common Reasons Why Your iPhone Keeps Randomly Turning Off
- Reinforcing Few-Step Diffusion Models via Shadow Posterior Matching
- Nvidia researchers argue huge AI context windows are a dead end
- Waymo Recalls Robotaxis After They Drove Into Closed Freeway Lanes
