The field of large language models is moving towards more efficient and scalable training methods. Researchers are exploring innovative techniques to reduce the computational resources required for training, such as memory-efficient backpropagation and optimal scaling methods. Additionally, there is a growing interest in understanding the underlying mechanisms that make diffusion language models highly efficient in limited-data constraints. Noteworthy papers include:

• Optimal Scaling Needs Optimal Norm, which discovers a unifying principle for optimal hyperparameter transfer across model and dataset sizes.
• GUIDE: Guided Initialization and Distillation of Embeddings, which introduces a novel distillation technique that forces the student to match the teacher in the parameter space, resulting in significant improvements in model quality.
• Boomerang Distillation Enables Zero-Shot Model Size Interpolation, which provides a simple and efficient way to generate fine-grained model families, reducing training costs and enabling flexible adaptation across deployment environments.