The field of large language models (LLMs) is rapidly advancing, with a focus on improving reasoning capabilities. Recent developments have led to the creation of novel benchmarks and evaluation frameworks, such as those assessing abstract reasoning, symbolic mathematics, and data-flow analysis. These advancements highlight the current limitations of LLMs in terms of true understanding and generalization, but also demonstrate the potential for significant improvements through innovative approaches like computational thinking and exchange of perspective prompting. Noteworthy papers include ASyMOB, which introduces a novel assessment framework for symbolic mathematics, and TimeHC-RL, which enhances LLMs' social intelligence through temporal-aware hierarchical cognitive reinforcement learning. Overall, the field is moving towards a deeper understanding of LLM reasoning capabilities and the development of more effective evaluation methods.
Advancements in Large Language Model Reasoning
Sources
Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark
Scaling up the think-aloud method
DSR-Bench: Evaluating the Structural Reasoning Abilities of LLMs via Data Structures
FABLE: A Novel Data-Flow Analysis Benchmark on Procedural Text for Large Language Model Evaluation
TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence
Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation
Computational Thinking Reasoning in Large Language Models
EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving
Exchange of Perspective Prompting Enhances Reasoning in Large Language Models
CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective
Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design