View a PDF of the paper titled The Post-Training Dilemma: Why We Should Rethink the Sequential SFT-RL Paradigm, by Xueyan Niu and 3 other authors
Abstract:This position paper argues that the sequential supervised fine-tuning followed by reinforcement learning (SFT-then-RL) pipeline—the de facto paradigm for large language model (LLM) post-training—has a fundamental property that cannot be fixed by hyperparameter tuning: SFT and RL optimize over nearly orthogonal parameter subspaces, so RL inevitably degrades the capabilities acquired during the preceding SFT phase. Under the Polyak-Lojasiewicz (PL) condition, which is empirically observed for over-parameterized networks, we show that RL increases the SFT loss by at least $\frac{\mu}{2}\mathrm{dist}(\theta_{\mathrm{RL}}, \Theta^*_{\mathrm{SFT}})^2$ for some $\mu>0$. Although the KL penalty can constrain distributional shift, it cannot correct the underlying gradient misalignment between the two objectives. Experiments on Qwen3-0.6B across four tasks confirm the theoretical predictions: on MATH-500, Alpaca, HH-RLHF, and SST-2, the SFT loss increases after the RL phase. We identify the underlying mechanism as persistent near-orthogonality between the gradients of the two objectives, paired with opposing landscape dynamics: SFT flattens the loss landscape while RL re-sharpens it. Based on our analysis, we propose three promising research directions: (1) joint optimization of the two objectives with gradient surgery to resolve conflicting updates; (2) landscape-aware scheduling that terminates RL training at the theoretically optimal point; and (3) characterizing the model scale threshold above which SFT becomes unnecessary.
Submission history
From: Xueyan Niu [view email]
[v1]
Mon, 12 Jan 2026 10:14:09 UTC (170 KB)
[v2]
Wed, 6 May 2026 07:45:21 UTC (176 KB)
[v3]
Thu, 19 Nov 2026 01:42:25 UTC (105 KB)
