arXiv:2611.25780v1 Announce Type: new
Abstract: Routing states in mixture-of-experts (MoE) models can distinguish correct from erroneous reasoning steps. If changing routing also changes a model’s step-level verification judgment, it is tempting to conclude that the judgment follows what routing encodes. We test this inference with RouteVerify, a controlled intervention study that replaces selected internal activations from one solution trace with activations from another, while keeping the evaluated input unchanged. Across nine MoE checkpoints, we compare three intervention interfaces: router logits, MoE outputs, and post-MoE residual states. We vary the layers and token positions at which activations are replaced. We measure changes in a verification margin that quantifies the model’s preference for INCORRECT over CORRECT. We also evaluate the same input with activations from correct versus erroneous steps to test whether the resulting judgment depends on the source step’s correctness. We report four linked results. First, probes on router logits separate correct from erroneous steps in eight checkpoints (AUROC 0.80-0.88). Second, replacing router logits across all routed layers changes the judgment in all nine checkpoints, and by more than a norm-matched random replacement. Third, this change rarely depends on whether the source step is correct. At one layer, routing from erroneous steps and from correct steps produces indistinguishable margins in all nine checkpoints. Under all-layer routing, their difference is resolved and positive in only one. Fourth, the same difference is positive for local residual replacements in eight of nine checkpoints, so the measure does detect such dependence where it exists. Correctness is therefore readable from routing, and routing replacements move the judgment, but the movement rarely follows correctness.
Trending
- New open model Inkling 2 Mini thinks in loops to solve problems with fewer words and still runs on a laptop
- RouteVerify: Routing Interventions for Step-Level Verification in Mixture-of-Experts Models
- US technology stocks suffer their worst weekly loss since the pandemic as AI doubts pile up
- [2609.04910] Fast Gauss Sums via Flash Attention
- 17 Best Gifts for Women in the Enshittocene (2026)
- Disentangling Monitor Hardware Artifacts and Smoothing Lag for Shift-Robust Intraoperative EEG and Hemodynamic Risk Calibration
- Learning Feature-Specific Timescales in Language Model Representations
- Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models
