[2606.03871] Visual Instruction Tuning Aligns Modalities through Abstraction

View a PDF of the paper titled Visual Instruction Tuning Aligns Modalities through Abstraction, by Luis Palacios and 3 other authors

View PDF
HTML (experimental)

Abstract:Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text. Yet it remains unclear how visual features are embedded within the layer-wise hierarchy of abstractions in the LLM backbone. Across a diverse set of vision-language architectures, we show that instruction tuning primarily serves as a bridge, embedding visual features directly into the intermediate semantic layers of the LLM, bypassing the early layers devoted to unimodal processing. With probing analyses and causal interventions, we show that these intermediate layers constitute the semantic core of vision-language processing and play a critical role in performance across a broad set of multimodal benchmarks. In addition, by comparing the geometry of semantically equivalent visual and textual representations, we find that fine-tuning extends and strengthens the existing abstraction phase, aligning visual features with pre-existing textual ones. Finally, we confirm the functional role of this localized alignment by restricting fine-tuning to intermediate layers alone: this strategy preserves the performance of full fine-tuning on vision-centric benchmarks while reducing training time. Our results suggest that multimodal integration is a localized phenomenon driven by the repurposing of the internal abstraction engine of the LLM.