The field of neural networks is moving towards more efficient and scalable optimization methods. Researchers are exploring new algorithms and techniques to improve the training process, including adaptive optimization methods and novel decay mechanisms. The importance of adapting to the structures in the problem and making algorithms agnostic to the scale of the problem is being highlighted. Additionally, there is a growing interest in learning and testing convex functions, particularly in high-dimensional spaces. Noteworthy papers include: AdamX, which proposes a novel exponential decay mechanism for the second-order moment estimate, improving the stability of training and generalization ability. AdamHD, which introduces a decoupled Huber decay regularization for language model pre-training, resulting in faster convergence and improved performance. ECPv2, which provides a scalable and theoretically grounded algorithm for global optimization of Lipschitz functions, outperforming state-of-the-art optimizers in high-dimensional problems.