The field of multimodal learning and vision-language models is rapidly advancing, with a focus on improving performance, efficiency, and adaptability. Recent research has explored innovative approaches to optimize image focus, adapt large language models to specific domains, and develop novel prompting mechanisms. Notably, the use of graph prompting, token coordinated prompt attention, and geometry-aware point cloud prompts has shown significant promise in enhancing model performance. Furthermore, advancements in out-of-distribution detection, uncertainty quantification, and open-world prompt tuning have expanded the capabilities of vision-language models. Overall, the field is moving towards more effective, efficient, and generalizable models that can handle complex, real-world tasks. Noteworthy papers include Zoomer, which introduces a novel visual prompting mechanism, and DeCLIP, which enhances CLIP's performance in open-vocabulary dense prediction tasks.
Advancements in Multimodal Learning and Vision-Language Models
Sources
Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
On the effectiveness of Large Language Models in the mechanical design domain
A Domain Adaptation of Large Language Models for Classifying Mechanical Assembly Components
GraphPrompter: Multi-stage Adaptive Prompt Optimization for Graph In-Context Learning
Always Skip Attention
Token Coordinated Prompt Attention is Needed for Visual Prompting
Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models
Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey
Seeing the Abstract: Translating the Abstract Language for Vision Language Models
Enhancing Target-unspecific Tasks through a Features Matrix
Panoramic Out-of-Distribution Segmentation
Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning
Calibrating Uncertainty Quantification of Multi-Modal LLMs using Grounding
MISE: Meta-knowledge Inheritance for Social Media-Based Stressor Estimation
GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model
Vision Graph Prompting via Semantic Low-Rank Decomposition
DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation
Adaptive Token Boundaries: Integrating Human Chunking Mechanisms into Multimodal LLMs
Flower Across Time and Media: Sentiment Analysis of Tang Song Poetry and Visual Correspondence
Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Does CLIP perceive art the same way we do?
Aesthetics Without Semantics
TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation