This research from JHU presents groundbreaking evidence that deep neural networks trained across vastly different tasks converge to shared, low-dimensional parametric subspaces. Fine-tuning using LoRA updates only a fraction of weights; this paper suggests that even these low-rank updates are redundant across different tasks.
- They show that a subspace learned from one set of tasks can be effectively applied to adapt models to completely different distributions with minimal performance loss.
- The paper provides mathematical analysis to explain why these subspaces emerge, linking them to the spectral properties of the model's Hessian and the pre-training data distribution.