The field of large language models is witnessing significant developments in attention mechanisms and context representation. Researchers are exploring new approaches to improve the scalability and effectiveness of these models, particularly in handling long-range dependencies and sparse attention patterns. The use of probabilistic frameworks, such as Bayesian attention mechanisms, and the incorporation of memory units and sparse caching are showing promising results in enhancing the performance of large language models. Furthermore, the development of new diagnostic frameworks and the study of emergence in attention patterns are advancing our understanding of how these models learn and generalize. Notable papers in this area include the proposal of AnchorAttention, which achieves superior speed and accuracy by efficiently identifying critical attention regions, and LoLA, which enables pass-key retrieval on up to 8K context lengths with a 4.6x smaller cache. Additionally, ATLAS, a long-term memory module, has been shown to surpass the performance of Transformers and recent linear recurrent models in language modeling and common-sense reasoning tasks.
Advances in Attention Mechanisms and Context Representation for Large Language Models
Sources
Scale-invariant Attention
Attention with Trained Embeddings Provably Selects Important Tokens
The emergence of sparse attention: impact of data distribution and benefits of repetition
TRACE for Tracking the Emergence of Semantic Representations in Transformers
Born a Transformer -- Always a Transformer?
Curse of High Dimensionality Issue in Transformer for Long-context Modeling
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
Structured Memory Mechanisms for Stable Context Representation in Large Language Models
AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity
Characterizing the Expressivity of Transformer Language Models
LoLA: Low-Rank Linear Attention With Sparse Caching
ATLAS: Learning to Optimally Memorize the Context at Test Time