arXiv:2611.26584v1 Announce Type: cross
Abstract: Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models’ own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($\Delta d’ = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.
Trending
- Represented but misread: Post-training changes how language models read out what they represent
- Mamba Flow Matching Neural Processes: Linear-Time Inference for Irregularly Observed Spatial Fields
- Cost-Aware Agentic Feature Optimization System for Fleet-Scale Ranking Models
- TRCF: Temporal-Reference Complementary Forecasting for Irregular Multivariate Time Series
- [2611.26515] Simple Private Noisy SGD is Minimax-Optimal under Population Strong Convexity
- OceanCoder: A Verification-Guided Large Language Model for Automated Simulation Modelling in Ocean Engineering
- Bayesian optimization on amortized search space
- [2611.26323] Predicting Surface Displacement Fields of Laser Powder Bed Fusion Parts Using STL-Derived Graphs and Part-Level Descriptors
