The field of visual reasoning and literacy is rapidly advancing, with a focus on improving the performance of multimodal large language models (MLLMs) in complex visual tasks. Recent developments have highlighted the importance of fine-grained visual evidence extraction, integration, and reasoning for genuine visual understanding and human-like analysis. Researchers are proposing novel benchmarks, such as VER-Bench and O-Bench, to evaluate MLLMs' ability to identify subtle visual clues and construct evidence-based arguments. Additionally, new methodologies like DRIVE-T and Charts-of-Thought are being introduced to enhance the construction and evaluation of assessment items for data visualization literacy. Noteworthy papers include Oedipus and the Sphinx, which proposes a dual optimization strategy to improve VLM performance, and Charts-of-Thought, which introduces a novel prompting technique to guide LLMs through systematic data extraction and analysis. Overall, the field is moving towards more nuanced and robust evaluations of visual activity recognition and visualization literacy, with a focus on developing more effective and efficient models for complex visual tasks.
Advancements in Visual Reasoning and Literacy
Sources
Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning
CrossSet: Unveiling the Complex Interplay of Two Set-typed Dimensions in Multivariate Data
ChartCap: Mitigating Hallucination of Dense Chart Captioning
SlideAudit: A Dataset and Taxonomy for Automated Evaluation of Presentation Slides
Tell Me Without Telling Me: Two-Way Prediction of Visualization Literacy and Visual Attention
Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models
DRIVE-T: A Methodology for Discriminative and Representative Data Viz Item Selection for Literacy Construct and Assessment
EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts
MisVisFix: An Interactive Dashboard for Detecting, Explaining, and Correcting Misleading Visualizations using Large Language Models
Charts-of-Thought: Enhancing LLM Visualization Literacy Through Structured Data Extraction
VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidence
Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering
Finding Needles in Images: Can Multimodal LLMs Locate Fine Details?
Critical Design Strategy: a Method for Heuristically Evaluating Visualisation Designs