The field of parameter-efficient fine-tuning is experiencing significant growth, with a focus on developing innovative methods to adapt large foundation models to various downstream tasks. Recent research has explored the integration of low-rank adaptation (LoRA) and mixture-of-experts (MoE) approaches to enhance performance and efficiency. Notably, the use of heterogeneous mixtures of adapters and dynamic expert libraries has shown promise in improving model generalization and reducing catastrophic forgetting. Furthermore, novel gradient surgery methods and model merging techniques have been proposed to address challenges in multi-task learning and out-of-domain generalization. These advancements have the potential to significantly impact the field, enabling more efficient and effective fine-tuning of large language models. Noteworthy papers include:

• Come Together, But Not Right Now, which proposes a progressive training strategy to boost low-rank adaptation.
• MoA: Heterogeneous Mixture of Adapters, which introduces a heterogeneous mixture-of-adapters approach for parameter-efficient fine-tuning.
• Dynamic Mixture of Progressive Parameter-Efficient Expert Library, which proposes a dynamic expert library for lifelong robot learning.
• Gradient Similarity Surgery, which introduces a novel gradient surgery method for multi-task deep learning.
• Merging Smarter, Generalizing Better, which proposes a layer-wise pruning task vector method for enhancing model merging on out-of-domain data.
• Harmonizing and Merging Source Models, which proposes a novel source model merging framework for CLIP-based domain generalization.