The field of neural networks is rapidly evolving, with a focus on developing innovative architectures and optimization techniques to improve performance and efficiency. One notable direction is the integration of concepts from biology and physics, such as the use of energy landscapes and thermodynamic entropy to understand the behavior of artificial neural networks. Another area of research is the development of novel optimization methods, including those that leverage gradient-free optimization and adaptive model merging. Furthermore, there is a growing interest in equivariant neural networks, which can preserve symmetry and improve performance in tasks such as image classification and fiber orientation distribution estimation. Noteworthy papers in this area include 'Architecture of Information', which explores the energetic nature of informational entropy, and 'Meta-Representational Predictive Coding', which introduces a biologically plausible framework for self-supervised learning. Additionally, 'Equivariant Spherical CNNs' demonstrates the effectiveness of rotationally equivariant neural networks in estimating fiber orientation distributions in neonatal diffusion MRI.
Advances in Neural Network Architectures and Optimization Techniques
Sources
Architecture of Information
Meta-Representational Predictive Coding: Biomimetic Self-Supervised Learning
Flexible Moment-Invariant Bases from Irreducible Tensors
Improving Equivariant Networks with Probabilistic Symmetry Breaking
A Proposal for Networks Capable of Continual Learning
ReLU Networks as Random Functions: Their Distribution in Probability Space
T-CIL: Temperature Scaling using Adversarial Perturbation for Calibration in Class-Incremental Learning
Robustness quantification and how it allows for reliable classification, even in the presence of distribution shift and for small training sets
Ancestral Mamba: Enhancing Selective Discriminant Space Model with Online Visual Prototype Learning for Efficient and Robust Discriminant Approach
Uncertainty Weighted Gradients for Model Calibration
Towards Understanding the Optimization Mechanisms in Deep Learning
Pareto Continual Learning: Preference-Conditioned Learning and Adaption for Dynamic Stability-Plasticity Trade-off
Autonomous Learning with High-Dimensional Computing Architecture Similar to von Neumann's
A Plasticity-Aware Method for Continual Self-Supervised Learning in Remote Sensing
NoProp: Training Neural Networks without Back-propagation or Forward-propagation
Gradient-free Continual Learning
Cram\'er--Rao Inequalities for Several Generalized Fisher Information
Equivariant Spherical CNNs for Accurate Fiber Orientation Distribution Estimation in Neonatal Diffusion MRI with Reduced Acquisition Time
Architect Your Landscape Approach (AYLA) for Optimizations in Deep Learning
On the Geometry of Receiver Operating Characteristic and Precision-Recall Curves
Reservoir Computing: A New Paradigm for Neural Networks
BECAME: BayEsian Continual Learning with Adaptive Model MErging
GMR-Conv: An Efficient Rotation and Reflection Equivariant Convolution Kernel Using Gaussian Mixture Rings