arXiv:2611.26592v1 Announce Type: cross
Abstract: Self-consistency certification treats repeated sampling from a language model as sequential evidence accumulation and stops when an anytime-valid e-process crosses a threshold, guaranteeing finite-sample Type I error control by Ville’s inequality. Prior constructions assume sampled rollouts are conditionally independent. We show this assumption fails in practice: reasoning streams are positively correlated through shared reasoning prefixes or across-prompt heterogeneity, changing not merely certificate efficiency but the hypothesis certified. Modeling dependence via a per-unit latent state (the random-effects / intraclass-correlation structure) induces exchangeable, equi-correlated increments with correlation $\rho$. Correlation splits the single null into a realized null about the drawn belief and a population null about average model behavior, which coincide only when $\rho=0$. We show: (i) the standard betting e-process remains anytime-valid for the realized null, not the population null; (ii) read as a population-null test, a suitable mixture has asymptotic crossing probability at least $\pi=P(\mu(Z)>0)$, potentially exceeding nominal $\epsilon$; (iii) the within-unit mean’s effective sample size saturates at $1/\rho$, so extra within-unit rollouts cannot restore population validity; and (iv) a prompt-level e-process restores population validity, with optimal allocation at one rollout per prompt under a pure rollout budget. Simulations and Claude Haiku 4.5 rollouts confirm this regime dependence: on GSM8K ($\hat{\rho}=0.435$, $\hat{\theta}=+0.96$) the gap is dormant, whereas on AIME ($\hat{\rho}=0.480$, $\hat{\theta}=-0.15$) the single-unit rule crosses on 24% of prompts at nominal $\epsilon=0.05$, while the prompt-level procedure declines (0.00). This correctness result identifies which reliability claim an anytime-valid certificate actually supports under dependence.
Trending
- HiSA: Sparsifying the Hidden Dimension of RoPE Attention in LLMs
- Realized versus Population Reliability: Anytime-Valid Self-Consistency Certification under Correlated Reasoning
- Learning from Context: Leave-One-Out Prediction in Jump Diffusions
- Represented but misread: Post-training changes how language models read out what they represent
- Mamba Flow Matching Neural Processes: Linear-Time Inference for Irregularly Observed Spatial Fields
- Cost-Aware Agentic Feature Optimization System for Fleet-Scale Ranking Models
- TRCF: Temporal-Reference Complementary Forecasting for Irregular Multivariate Time Series
- [2611.26515] Simple Private Noisy SGD is Minimax-Optimal under Population Strong Convexity
