arXiv:2611.25597v1 Announce Type: new
Abstract: Large language models (LLMs) have shown promise as predictors across various tasks, benefiting from their ability to use information from unstructured context. However, when predictions serve as parameters of downstream optimization problems, higher prediction accuracy does not necessarily translate into lower decision costs. We propose decision-focused reinforcement learning (DFRL), a framework for post-training LLMs to predict optimization parameters using negative downstream decision regret as reward. DFRL trains LLM predictors to minimize the costs of the decisions induced by their predictions, without requiring downstream convexity or differentiability, and integrates readily into existing RL post-training frameworks. To address expensive reward evaluation, we develop two approaches. Heuristic warm-starting uses inexpensive approximate rewards during an initial phase before switching to refinement with exact rewards. Reward correction combines inexpensive estimates with randomized exact evaluations, preserving the expected policy gradient while reducing the expected number of exact solves per update. Experiments on three synthetic optimization problems demonstrate improvements over prediction-loss baselines, while a larger knapsack experiment shows faster early improvement under short training budgets. On inventory planning with H&M transactions and product descriptions, our best configuration, in which the LLM reasons before predicting, reduces regret by 31.4% relative to the SFT baseline and 12.4% relative to the strongest evaluated conventional forecasting baseline.
Trending
- Text-Aware, Vision-Blind: How Multimodal Image Editing Systems Handle Unsafe Inputs
- Decision-Focused Reinforcement Learning for LLM Post-Training
- [2611.26476] Anticipating Future Build Results with BuildPredictor
- Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models
- Exact Bayesian layers via characteristic functions
- Few-Shot Cross-City Traffic Prediction with Conditional Diffusion
- Simulation-pretrained autoencoders versus PCA for compressing multimode fibre transmission matrices
- Design and Evaluation of Epira: An Automated Interview System for Case Investigation and Contact Tracing
