arXiv:2611.25594v1 Announce Type: new
Abstract: Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action’s outcomes. We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones. On a simulated robot push with enumerated outcome distributions, this objective takes a model that is accurate on unseen layouts (hazard error 2.4 percentage points, pp) and moves 33 to 43 pp of its probability onto outcomes that none of a condition’s enumerated branches produces. The cause is the objective, not the comparisons: its optimum, computed before training, predicts where the model ends up, and a likelihood that knows the judge’s rule learns from the same comparisons without the shift. The shift is robust: it recurs on unseen conditions, under five candidate laws and a three-fold denser enumeration, and about 29 pp remain without the physically impossible events. Its cost is conditional: with those events it inflates hazard probabilities, and a threshold inherited from the pre-feedback model costs about 3 pp of utility per decision on trained and 1 to 2 pp on unseen conditions, through false rejects; without them the utility loss averages below 1 pp over six seeds, though a small regret in action choice remains. Audits of readout, response and decision trace the effect from probabilities to actions.
Trending
- [2611.26476] Anticipating Future Build Results with BuildPredictor
- Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models
- Exact Bayesian layers via characteristic functions
- Few-Shot Cross-City Traffic Prediction with Conditional Diffusion
- Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices
- Design and Evaluation of Epira: An Automated Interview System for Case Investigation and Contact Tracing
- [2611.26291] Who Moved My $Z$ Boson? Revealing Resonance Features Inside a Neural Network
- [2611.25570] Reinforcement-Aligned Probabilistic Weather Forecasting with Discrete Tokens
