Advances in Efficient Distributed Learning

The field of distributed learning is moving towards developing more efficient and scalable algorithms that can handle large-scale data and reduce communication costs. Researchers are exploring new techniques such as adaptive client selection, sparse quantization, and variance reduction to improve the performance of distributed learning models. Notably, the use of asynchronous algorithms and dynamic topologies is becoming increasingly popular.

Some noteworthy papers in this area include: Harmonized Gradient Descent for Class Imbalanced Data Stream Online Learning, which introduces a novel algorithm for handling imbalanced data streams. FedSODA, which proposes a resource-efficient federated fine-tuning framework for large language models. Communication-Efficient Distributed Asynchronous ADMM, which reduces communication overhead in distributed optimization. SparseLoCo, which achieves extreme compression ratios in communication-efficient LLM pre-training.

Sources

Harmonized Gradient Descent for Class Imbalanced Data Stream Online Learning

Communication-Efficient Distributed Asynchronous ADMM

Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings

FedSODA: Federated Fine-tuning of LLMs via Similarity Group Pruning and Orchestrated Distillation Alignment

Communication-Efficient Federated Learning with Adaptive Number of Participants

SSSP-Del: Fully Dynamic Distributed Algorithm for Single-Source Shortest Path

Cooperative SGD with Dynamic Mixing Matrices

Federated Learning based on Self-Evolving Gaussian Clustering

Measures of Overlapping Multivariate Gaussian Clusters in Unsupervised Online Learning

Jointly Computation- and Communication-Efficient Distributed Learning

Communication Efficient LLM Pre-training with SparseLoCo

Built with on top of