Federated Learning for Large Language Models

The field of federated learning is moving towards more efficient and privacy-preserving methods for fine-tuning large language models. Recent developments have focused on addressing the challenges of computational and communication costs, as well as the limitations of accessing internal model information. Innovative approaches have emerged, including layer pruning, quantization, and adaptive distillation, which enable the deployment of large language models on resource-constrained devices. These advancements have the potential to improve model accuracy and reduce memory usage, making federated learning more accessible and effective. Notable papers in this area include:

  • Memory-Efficient Federated Fine-Tuning of Large Language Models via Layer Pruning, which proposes a novel pruning paradigm that significantly improves model accuracy and reduces memory usage.
  • Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices, which introduces a system designed to enable federated fine-tuning of MoE-based LLMs on consumer-grade GPUs.
  • Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data, which proposes a unified benchmarking framework and a framework designed to address multi-domain non-IID challenges in both homogeneous and heterogeneous settings.

Sources

On the Evolution of Federated Post-Training Large Language Models: A Model Accessibility View

Memory-Efficient Federated Fine-Tuning of Large Language Models via Layer Pruning

Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devices

Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data

Built with on top of