The field of Transformer research is moving towards developing more efficient attention mechanisms to improve performance and reduce computational costs. Recent advances focus on modifying existing attention mechanisms to better handle global information modeling and reduce quadratic complexity. One notable direction is the incorporation of gated linear units (GLUs) into attention mechanisms, which has shown great potential in enhancing model performance. Additionally, researchers are exploring ways to improve linear attention, which has similar formulation to softmax attention but suffers from performance degradation due to neglect of magnitude information. Another approach is to introduce new attention mechanisms inspired by techniques from other fields, such as numerical simulations, to achieve linear time and memory complexity. Noteworthy papers include:

• Masked Gated Linear Units, which introduces an efficient kernel implementation for GLUs, achieving significant inference-time speed-up and memory efficiency.
• Magnitude-Aware Linear Attention, which modifies linear attention to incorporate magnitude information, achieving strong results on multiple tasks.
• Multipole Attention Neural Operator, which computes attention in a distance-based multiscale fashion, maintaining a global receptive field and achieving linear time and memory complexity.