arXiv:2611.17772v1 Announce Type: new
Abstract: Knowledge distillation (KD) is a highly effective paradigm for transferring capabilities from large language models (LLMs) to smaller, more specialized student models. In commercial deployment where frontier models are restricted behind black-box APIs, sequence-level distillation remains a potent vector for competitors to clone proprietary capabilities at a fraction of the development cost. Existing defenses mostly rely on surrogate models to manipulate next-token sampling, which increases computational overhead and real-time inference latency. In this work, we propose a lightweight defense that dynamically injects style-steering vectors into the teacher model’s activations upon malicious queries. This framework offers exceptional flexibility, allowing hosts to customize target defensive styles and dynamically blend them to maximize stylistic variance. Crucially, this inference-time intervention preserves semantic utility and fluency during malicious queries, misleading adversaries into believing that they are extracting pristine teacher outputs, while standard consumers continue to receive original, unmodified responses. Our experiments demonstrate that this framework successfully disrupts student learning, achieving substantial anti-distillation performance while incurring negligible overhead and preserving text naturalness. Our code is available at https://anonymous.4open.science/r/distill-resist-38C6/.
Trending
- What Data Storage Looked Like In The Early 2000s Vs. Today
- DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention
- Z2RAM Layers Promise Denser SRAM Memory for AI
- Why Deep Discounts On Used Apple AirPods Just Aren’t Worth It
- [2106.14565] Variance Reduction for Matrix Computations with Applications to Gaussian Processes
- The Pros And Cons Of Using A Dumb Phone Over A Smartphone In 2026
- [2611.25921] A Lightweight Hybrid Approach for Milling Force Reconstruction from Internal Machine Signals
- What Are The Actual Differences Between Cheap And Expensive Sound Systems?
