The field of Mixture-of-Experts (MoE) models is rapidly advancing, with a focus on improving efficiency, scalability, and performance. Recent research has explored various techniques to optimize MoE models, including mixed-precision quantization, on-the-fly inference, and expert allocation methods. These innovations aim to address the challenges of deploying MoE models on resource-constrained devices and in multi-tenant environments. Notably, the importance of feedforward networks in transformer models has been highlighted, and the benefits of fine-grained experts in boosting expressivity have been demonstrated. Additionally, new architectures and frameworks, such as UMoE and PT-MoE, have been proposed to unify attention and feedforward networks, and to integrate mixture-of-experts into prompt tuning, respectively. Noteworthy papers include MxMoE, which introduces a mixed-precision optimization framework for MoE models, and FloE, which proposes an on-the-fly MoE inference system for memory-constrained GPUs. LoRA-SMoE is also notable for its sensitivity-driven expert allocation method, which enables efficient fine-tuning of MoE models. QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration is another significant work, as it addresses the challenge of serving multiple fine-tuned MoE-LLMs on a single GPU.