The field of large language models is moving towards more efficient and cost-effective solutions. Recent developments focus on reducing the memory footprint and computational demands of these models, making them more suitable for local deployment and edge devices. Innovations in model architecture, such as hybrid approaches and sparse structures, are being explored to improve performance while minimizing costs. Additionally, techniques like quantization, caching, and compression are being developed to optimize model inference. Notable papers in this area include A3D-MoE, which proposes a 3D heterogeneous integration system to enhance memory bandwidth and reduce energy consumption, and SmallThinker, which introduces a family of efficient large language models natively designed for local deployment. Other noteworthy papers include HCAttention, which enables extreme KV cache compression via heterogeneous attention computing, and Falcon-H1, which presents a family of hybrid-head language models redefining efficiency and performance.
Efficient Large Language Models
Sources
A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration
Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding
HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
CaliDrop: KV Cache Compression with Calibration
FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression
SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
MemShare: Memory Efficient Inference for Large Reasoning Models through KV Cache Reuse
Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
KLLM: Fast LLM Inference with K-Means Quantization
Beyond the Cloud: Assessing the Benefits and Drawbacks of Local LLM Deployment for Translators