The field of large language models is moving towards developing more human-like representations, with a focus on advancing the capabilities of these models to better align with human cognition and conceptual understanding. Researchers are exploring various approaches, including the use of symbolism, comprehensive evaluation frameworks, and novel benchmarking methods, to improve the performance and reliability of large language models. Notably, studies have shown that instruction-finetuning and larger dimensionality of attention heads can lead to more human-aligned representations, while multimodal pretraining and parameter size have limited bearing on alignment. Furthermore, the development of diagnostic frameworks, such as Benchmark Profiling, is providing a more transparent and systematic way to evaluate the capabilities of large language models. Some papers are particularly noteworthy, including: Uncovering the Computational Ingredients of Human-Like Representations in LLMs, which identifies key computational ingredients for advancing LLMs towards human-like representations. Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks, which introduces a diagnostic framework to decompose benchmark performance into cognitively grounded abilities.
Large Language Models and Human-Like Representations
Sources
CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and Task
Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning