arXiv:2611.17879v1 Announce Type: new
Abstract: Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.
Trending
- Nothing’s AI Device Debuts December 8, And It Looks Like Two Vapes Stuck Together
- Why We Should Rethink the Sequential SFT-RL Paradigm
- Chips can survive the space environment. Can their financing?
- Prescribed Bayesian Efficiency under Lifetime Adjustment Costs
- Robot Videos: Morphing Robots, Dancing Humanoids, More
- You Can Reveal The Full GTA 6 Map In Minutes Using This Simple Plane Trick
- State-Space Compression for AI-Based Radio Resource Management of Vehicular Services
- The 23 Best Early Black Friday Deals: Our Ultimate Picks (2026)
