The field of large language models (LLMs) is seeing significant advancements in optimization techniques, with a focus on improving memory efficiency and convergence rates. Researchers are exploring new methods to mitigate the limitations of traditional optimization algorithms, such as the reliance on standard isotropic steepest descent techniques. Novel approaches, including subspace-aware moment-orthogonalization and adaptive preconditioners, are being proposed to address these challenges. These innovations have the potential to accelerate convergence, enhance stability, and reduce memory requirements, making LLM training more efficient and accessible. Noteworthy papers in this area include:

• SUMO, which proposes a subspace-aware moment-orthogonalization optimizer that achieves improved convergence rates and reduced memory requirements.
• Purifying Shampoo, which investigates the heuristics of the Shampoo algorithm and proposes a principled approach to removing these heuristics.
• Leveraging Coordinate Momentum in SignSGD and Muon, which introduces memory-optimized zero-order optimization methods for fine-tuning LLMs.
• Adaptive Preconditioners Trigger Loss Spikes in Adam, which identifies the mechanism behind loss spikes in Adam and provides insights into the underlying causes of instability.