The field of distributed learning is moving towards more scalable and efficient methods, with a focus on reducing communication overhead and improving convergence rates. Recent works have explored various techniques, including relaxed global communication, alternating low-rank updates, and cyclical updates, to achieve these goals. These innovations have the potential to improve the performance of large language models and other applications. Noteworthy papers include Pier, which proposes an efficient and scalable optimizer for large language model pretraining, and ADF-LoRA, which introduces a decentralized federated learning method with improved stability and convergence. Additionally, CycleSL presents a novel split learning framework that enhances scalability and performance, while Mitigating Participation Imbalance Bias in Asynchronous Federated Learning provides a theoretical analysis and proposes methods to address heterogeneity amplification in asynchronous federated learning.
Scalable and Efficient Distributed Learning
Sources
Pier: Efficient Large Language Model pretraining with Relaxed Global Communication
ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning
CycleSL: Server-Client Cyclical Update Driven Scalable Split Learning
Mitigating Participation Imbalance Bias in Asynchronous Federated Learning
Towards a future space-based, highly scalable AI infrastructure system design
Federated Learning Framework for Scalable AI in Heterogeneous HPC and Cloud Environments
Row-stochastic matrices can provably outperform doubly stochastic matrices in decentralized learning
Accelerating Wireless Distributed Learning via Hybrid Split and Federated Learning Optimization
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning
On the Limits of Momentum in Decentralized and Federated Optimization
Communication-Efficient Learning for Satellite Constellations