Advances in Efficient Deep Learning Models

The field of deep learning is moving towards more efficient models that can achieve state-of-the-art performance while reducing computational resources and memory requirements. Recent developments have focused on improving the training and pruning of large language models and vision transformers. Techniques such as eigenspectrum analysis, preconditioned gradient descent, and structured pruning have shown promising results in diagnosis, optimization, and compression of deep neural networks. Noteworthy papers include:

  • Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias, which proposes a method to mitigate aspect ratio bias in eigenspectrum analysis.
  • NysAct, a scalable first-order gradient preconditioning method that strikes a balance between state-of-the-art first-order and second-order optimization methods.
  • Olica, a pruning framework for large language models that eliminates the need for retraining.
  • Diversity-Guided MLP Reduction, a method to reduce the parameters of large vision transformers with negligible performance degradation.
  • SparseSSM, the first training-free pruning framework for state-space language models.

Sources

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation

Olica: Efficient Structured Pruning of Large Language Models without Retraining

Diversity-Guided MLP Reduction for Efficient Large Vision Transformers

Data-Efficient Challenges in Visual Inductive Priors: A Retrospective

SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot

Built with on top of