The field of artificial intelligence is moving towards more efficient architectures and adaptive computation. Researchers are exploring ways to reduce computational costs while maintaining performance, such as pruning, quantization, and dynamic routing. Another trend is the development of adaptive models that can adjust their computation based on input complexity, latency constraints, and hardware capabilities. These advancements have the potential to enable more widespread adoption of AI in resource-constrained devices and applications. Noteworthy papers include DeepCoT, which proposes a redundancy-free encoder-only model for real-time inference on data streams, and AdaPerceiver, which introduces a transformer architecture with unified adaptivity across depth, width, and tokens. Other notable papers, such as IDAP++ and RefTr, demonstrate significant improvements in model compression and vascular tree analysis, respectively. Overall, the field is shifting towards more efficient, adaptive, and scalable AI models.
Efficient Architectures and Adaptive Computation in AI
Sources
DeepCoT: Deep Continual Transformers for Real-Time Inference on Data Streams
AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
In Search of Goodness: Large Scale Benchmarking of Goodness Functions for the Forward-Forward Algorithm
EVCC: Enhanced Vision Transformer-ConvNeXt-CoAtNet Fusion for Classification
Forecasting AI Time Horizon Under Compute Slowdowns
Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning
Rethinking Vision Transformer Depth via Structural Reparameterization
IDAP++: Advancing Divergence-Based Pruning via Filter-Level and Layer-Level Optimization
RefTr: Recurrent Refinement of Confluent Trajectories for 3D Vascular Tree Centerline Graphs
Subjective Depth and Timescale Transformers: Learning Where and When to Compute
Frequency-Aware Token Reduction for Efficient Vision Transformer
On the Origin of Algorithmic Progress in AI
Mechanisms of Non-Monotonic Scaling in Vision Transformers