The field of large language models (LLMs) is rapidly evolving, with a focus on improving their ability to acquire and retain knowledge. Recent developments have centered around enhancing the efficiency and effectiveness of knowledge probing, adaptation, and retention in LLMs. Notably, researchers have explored innovative methods to probe LLMs' knowledge without requiring computationally expensive forward passes, and have proposed novel architectures and techniques to adapt LLMs to new tasks and domains. Additionally, there is a growing interest in developing more expressive and parameter-efficient fine-tuning methods, such as low-rank adaptation and its variants.

Some noteworthy papers in this area include: Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings, which proposes a method to predict LLM knowledge using pre-trained embedding models. LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning, which introduces a highly efficient architecture expansion method for continual visual instruction tuning. DySK-Attn: A Framework for Efficient, Real-Time Knowledge Updating in Large Language Models via Dynamic Sparse Knowledge Attention, which enables LLMs to efficiently integrate real-time knowledge from a dynamic external source. Learning Facts at Scale with Active Reading, which proposes a framework for training models to study a given set of material with self-generated learning strategies. Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation, which proposes an asymmetric low-rank adaptation architecture for medical lay language generation.