arXiv:2611.26075v1 Announce Type: cross
Abstract: Understanding a language model’s decision requires knowing how its internal computations affect its preference between answers. Gradient-based methods estimate this influence by predicting the effect of changing those computations. In mixture-of-experts (MoE) models, replacing one expert can change which experts later layers select. Those switches can alter the answer scores in ways that derivatives do not capture. We introduce a predictor that uses the model’s own expert outputs and selection rules to estimate these extra effects, then adds them to a second-order prediction. Across questions from three benchmarks, our method reduces the error in predicting answer-score changes by 60\% on Qwen3.6 and 47\% on Gemma4, relative to a multistep baseline using gradients and second derivatives.
Trending
- Review: Godzilla Minus Zero is everything we could want in a sequel
- Amazon is complaining about NYC Council on shipping labels, too
- This Built-In Windows Tool Can Tell If Your DirectX Has A Problem
- Understanding MoE Decisions Beyond Gradients
- FBI And USDA Were Reportedly Targeted In Back-To-Back Cyberattacks
- NariQ: Sub-3-Bit Quantization for Long-Horizon Agentic Tasks
- How Long Can Fiber Optic Cables Be Before They Lose Signal?
- [2611.23445] Why Sub-4-Bit Vision CNNs Collapse and Where Mixed-Precision Recovery Stops
