arXiv:2611.25989v1 Announce Type: new
Abstract: Continual reinforcement learning (CRL) requires models to acquire new capabilities while retaining performance on previous tasks. A stability-oriented perspective common in conventional continual learning emphasizes knowledge preservation and interference mitigation. Complementing this perspective, we revisit CRL for vision-language models through policy improvement, a fundamental principle of RL, to preserve acquired performance gains. Using the performance difference lemma, we characterize historical performance changes through the current policy’s expected advantages on historical task distributions, motivating replay of historical prompts with the current policy. To allocate limited replay memory, we propose Contrastive Likelihood Replay (CoLiR), which contrasts likelihood gains on correct responses from the final task policy with likelihood reductions on incorrect responses from the initial policy. Our analysis links correct-response likelihood gains to reference-policy advantages under an idealized KL-regularized optimal update and provides an endogenous-reward interpretation for general policy updates. CoLiR estimates scores from finite training rollouts and cached log-likelihoods, selects prompts at task boundaries, and replays them uniformly during subsequent training, with KL regularization restricted to replayed prompts. Memory construction only requires a small amount of additional cross-policy scoring but no additional response generation. Across three Qwen3-VL model sizes on MRCL, CoLiR achieves the highest mean final accuracy and mean task accuracy among the compared methods while maintaining competitive new-task performance.
Trending
- Waymo Recalls Robotaxis After They Drove Into Closed Freeway Lanes
- [2603.24974] The Value of Information in Resource-Constrained Pricing
- Silicon Valley’s growing obsession with AI doomsday is turning parts of the tech industry toward cult-like devotion
- The Pros And Cons Of Using A VPN Over Encrypted DNS
- Policy Improvement-Guided Replay for Continual Reinforcement Learning
- Is selling an AI version of your face to this microdrama startup the future of Hollywood?
- Ways You Can Help Your Fitbit’s Battery Last Longer
- Attributing Gains in VLA Models Through Controlled Ablation
