The field of Large Language Models (LLMs) is moving towards more efficient and private fine-tuning methods, particularly in federated learning settings. Recent developments have focused on reducing communication costs, improving model adaptation, and mitigating non-IID data challenges. Notable advancements include novel low-rank adaptation techniques, adaptive federated fine-tuning frameworks, and sparse zeroth-order optimization methods. These innovations have demonstrated significant improvements in performance, efficiency, and robustness. Noteworthy papers include: SEMFED, which achieves an 80.5% reduction in communication costs while maintaining model accuracy above 98%. DenseLoRA, which enhances parameter efficiency and achieves superior performance compared to existing low-rank adaptation approaches. AFLoRA, which provides a practical solution for efficient LLM adaptation in heterogeneous environments. EcoLoRA, which significantly reduces communication overhead without compromising performance. PoLAR, which yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. DiaBlo, which eliminates the need for low rank matrix products and achieves stable and robust convergence. Meerkat, which achieves remarkable communication efficiency and effectively mitigates Non-IID data challenges.
Efficient Federated Learning for Large Language Models
Sources
SEMFED: Semantic-Aware Resource-Efficient Federated Learning for Heterogeneous NLP Tasks
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models
Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation
Advantageous Parameter Expansion Training Makes Better Large Language Models
AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models
Reconciling Hessian-Informed Acceleration and Scalar-Only Communication for Efficient Federated Zeroth-Order Fine-Tuning
WeightLoRA: Keep Only Necessary Adapters
Memory-Efficient Split Federated Learning for LLM Fine-Tuning on Heterogeneous Mobile Devices
FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
PoLAR: Polar-Decomposed Low-Rank Adapter Representation
DiaBlo: Diagonal Blocks Are Sufficient For Finetuning
Mitigating Non-IID Drift in Zeroth-Order Federated LLM Fine-Tuning with Transferable Sparsity