Not All Directions Matter: Toward Structured and Task-Aware Low-Rank Adaptation

Abstract

Key Results at a Glance

Method Overview

We identify two fundamental, unaddressed shortcomings in the LoRA paradigm. Semantic drift stems from allocating a limited parameter budget uniformly across all low-rank update directions, assuming each direction is equally important. Structural incoherence arises from adapting each layer independently, disregarding the compositional structure of deep Transformers.

Figure 1. Architectural comparison between LoRA and StructLoRA. Standard LoRA (left) applies uniform low-rank updates. StructLoRA (right) introduces an Information Bottleneck (IB) Filter (Stage 1) and a Graph-based Coordination mechanism (Stage 2). Both operate only during training and are removed at inference, preserving LoRA's zero-latency efficiency.

Stage 1: Information Bottleneck-Guided Directional Filtering

LoRA spreads a small budget over $r$ directions and treats them the same. In practice, only a few directions help predict the label; others carry nuisance variation. We gate the $r$ rank-one directions with a learnable mask $\mathbf{m} \in [0,1]^r$ and form the filtered update:

$\Delta \tilde{W} = A\,\text{diag}(\mathbf{m})\,B$

We learn $\mathbf{m}$ by an Information Bottleneck objective that rewards dependence on labels while penalizing spurious dependence on inputs, effectively raising the signal-to-noise ratio of the update.

Stage 2: Graph-Based Layer Coordination

We view the network as a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with one node per layer, where the node feature is the flattened filtered update. We connect adjacent layers and add semantic edges between layers with highly aligned gradients. A shallow GNN with residual connections propagates and refines the filtered updates, encouraging smoother adaptation trajectories across the model's depth. This can be formalized as Laplacian smoothing that provably reduces inter-layer drift energy.

Crucially, both the IB filter and the GNN coordinator are training-only — they are discarded at inference, so latency stays identical to vanilla LoRA.

Main Results

Table 1. Main comparison across language, vision, and multimodal benchmarks (~0.5–1% trainable parameters).

Table 2. Head-to-head comparison on the GLUE benchmark (RoBERTa-base).

Performance in Challenging Regimes

StructLoRA shines brightest where it matters most — under low-rank and low-data conditions.

Table 3. Performance under varying rank budgets.

Table 4. Performance under limited supervision (few-shot learning), rank $r=8$.

Ablation Studies

Both the IB filter and the GNN coordinator are essential. Removing either degrades performance, and removing both collapses StructLoRA back to standard LoRA.

Table 5. Module-wise ablation of StructLoRA.

Figure 2. Analysis of filtering strategies. Our IB-guided filter consistently outperforms heuristic alternatives (Random Masking and Top-$k$ Norm) across NLP, Vision, and Multimodal tasks, confirming that the magnitude of an update direction is a poor proxy for its semantic relevance.

Visualizing Structural Coherence

Figure 3. Visual attention comparison (Grad-CAM). LoRA (top) produces diffuse attention across background regions, while StructLoRA (bottom) focuses on semantically relevant areas such as the deer's head and the boat's body.

Figure 4. Layer-wise cosine similarity of updates. StructLoRA (left) induces a coherent block-diagonal structure, while LoRA (right) exhibits noisy, fragmented inter-layer patterns — confirming the semantic drift problem that our framework resolves.

Figure 5. Accuracy vs. Sequence Length (LLaMA-13B on BoolQ). StructLoRA's advantage widens under longer contexts.

Figure 6. SVD Coverage vs. Rank. StructLoRA achieves broader singular-value coverage, especially at low ranks.

BibTeX

If you find our work useful, please consider citing: