arXiv:2611.17876v1 Announce Type: new
Abstract: A temporary optimization intervention can increase validation loss when it ends yet reduce it at the final training budget. We demonstrate this with temporary weight decay in Muon-trained decoder-only Transformers with 162M, 405M, 1.4B, and 3.4B parameters. For models up to 1.4B parameters, moving a fixed-duration decay window from early to late training reverses its final effect. At 162M, changing window duration alters final loss even at approximately matched total shrinkage. Most strikingly, a schedule selected at 162M and transferred without retuning to 1.4B is 0.26 nats worse than no decay when removed, overtakes it only after 73% of training, and ultimately finishes 0.028 nats better while also outperforming the best sampled constant-decay baseline. Early decay leaves a persistent weight-norm deficit, which keeps relative steps larger after the intervention ends. An approximate norm-dynamics recurrence captures the slow convergence of these relative steps, and fixed-norm replays of the measured trajectories reproduce the performance ordering and a similar loss difference. At 162M and 405M, norm-reset experiments show that smaller weight norms at the end of the decay window explain most of the final loss improvement. A separate Gram intervention exhibits related post-removal norm and relative-step dynamics. These results show that temporary optimization schedules can have delayed effects long after removal and should therefore be evaluated at the intended final training budget, not only at removal.
Trending
- Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels
- YouTube is reportedly preparing to settle social media addiction claims for $3 billion
- GTA 6 Made $3.79 Billion In Its First Day, But Rockstar’s Workers Still Lack Job Security
- Antom 3-in-1 Transformer: A Payment Foundation Model
- A leaky AI supply chain is exposing secret user data at an alarming rate
- Forecasting LLM Activations Before the First Answer Token
- Astronomers watched a star spiral to its death inside a giant companion
- OpenAI makes it much harder for attackers to extract how its latest AI models think
