Similarity Systems and Associative Memory

The field of similarity systems and associative memory is witnessing a significant shift towards more compositional and adaptive approaches. Researchers are moving away from traditional hash-function engineering and instead focusing on witness-overlap structures and modular construction of similarity systems. This new paradigm enables the development of more robust and efficient similarity systems that can be combined and reused in various contexts. The introduction of adaptive similarity measures and the integration of Hopfield networks with Transformer architectures are also notable trends in this area. Noteworthy papers include: REWA, which introduces a general theory of similarity based on witness-overlap structures and provides a principled foundation for similarity systems. Adaptive Hopfield Network, which proposes a novel mechanism for learning adaptive similarities and achieves state-of-the-art performance across diverse tasks. On the Role of Hidden States of Modern Hopfield Network in Transformer, which establishes a more generalized form of correspondence between Hopfield networks and Transformers and improves the nature of attention weights.

Sources

REWA: Witness-Overlap Theory -- Foundations for Composable Binary Similarity Systems

Popularity Bias Alignment Estimates

Adaptive Hopfield Network: Rethinking Similarities in Associative Memory

On the Role of Hidden States of Modern Hopfield Network in Transformer

Built with on top of