arXiv:2611.17784v1 Announce Type: new
Abstract: Parameter-efficient fine-tuning (PEFT) aims to preserve adaptation quality while reducing training costs. We propose SCALAR, which achieves performance comparable to LoRA with substantially fewer trainable parameters, lower peak training memory, and higher throughput. SCALAR constructs a fixed input basis from task activations and assigns a random output dictionary to each input direction. Fine-tuning learns only layer-specific scalar coefficients, reducing gradient and optimizer-state storage. Grouped computation aggregates output atoms before applying the adapter to token activations, while dictionary sharing across compatible layers reduces fixed-basis storage. Experiments on commonsense reasoning, mathematical reasoning, code generation, and knowledge evaluation after instruction tuning demonstrate effective adaptation at small trainable budgets. On commonsense reasoning, SCALAR achieves accuracy comparable to LoRA with about 4% of its trainable parameters. System measurements on Qwen2.5-7B show 20% lower peak training memory and 6% higher throughput than LoRA. The combination of adaptation quality, a compact trainable state, and training efficiency makes SCALAR a promising approach to resource-constrained fine-tuning.
Trending
- 17 Best Gifts for Women in the Enshittocene (2026)
- Disentangling Monitor Hardware Artifacts and Smoothing Lag for Shift-Robust Intraoperative EEG and Hemodynamic Risk Calibration
- Learning Feature-Specific Timescales in Language Model Representations
- Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models
- [2606.16694] When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers
- Bytedance argues most AI training phases are unnecessary and backs it up with a competitive open model
- PosteriorBench: Evaluating Behavior Posterior Sampling in Language Models
- Source-free Proxy Anchor Concept Erasure for MLLMs
