The field of deep learning is moving towards more efficient and effective optimization techniques. Recent studies have focused on developing new methods that can improve the training process of deep neural networks. One direction is the development of adaptive optimization methods that can adapt to the specific needs of the problem at hand. Another direction is the use of spectral gradient methods, which have been shown to be effective in certain regimes. Additionally, there is a growing interest in developing more robust and reliable optimization methods that can handle non-convex and non-smooth functions. Notable papers include: The paper 'Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance' which proposes a minimal yet representative setting of next-token prediction, where the authors can provably show faster convergence of coordinate-wise algorithms. The paper 'Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning' which introduces a preconditioning procedure that accelerates Newton-Schulz convergence and reduces its computational cost. The paper 'Gradient Descent with Provably Tuned Learning-rate Schedules' which develops novel analytical tools for provably tuning hyperparameters in gradient-based algorithms that apply to non-convex and non-smooth functions.