The field of medical research is moving towards increased use of multimodal data, including images, text, and patient records, to improve disease diagnosis and treatment. This shift is driven by the development of new models and frameworks that can integrate and analyze multiple types of data. One area of focus is the use of large language models to improve the accuracy and efficiency of clinical tasks, such as diagnosis and patient data analysis. Another area of research is the development of multimodal models that can incorporate image and text data to improve disease diagnosis and treatment outcomes. Overall, the field is moving towards a more integrated and holistic approach to medical research, with a focus on using multiple types of data to improve patient outcomes. Noteworthy papers include: Combating the Bucket Effect: Multi-Knowledge Alignment for Medication Recommendation, which introduces a new framework for medication recommendation that integrates multiple types of knowledge data. Temporal Entailment Pretraining for Clinical Language Models over EHR Data, which proposes a novel pretraining objective for clinical language models that takes into account the temporal nature of patient data. HRScene, which introduces a new benchmark for high-resolution image understanding in the medical domain.
Advances in Multimodal Medical Research
Sources
Combating the Bucket Effect:Multi-Knowledge Alignment for Medication Recommendation
Temporal Entailment Pretraining for Clinical Language Models over EHR Data
HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?
Large Language Model Empowered Privacy-Protected Framework for PHI Annotation in Clinical Notes
Use of Metric Learning for the Recognition of Handwritten Digits, and its Application to Increase the Outreach of Voice-based Communication Platforms
HoloDx: Knowledge- and Data-Driven Multimodal Diagnosis of Alzheimer's Disease
CLIP-KOA: Enhancing Knee Osteoarthritis Diagnosis with Multi-Modal Learning and Symmetry-Aware Loss Functions
BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports
Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records
Leveraging Generative AI Through Prompt Engineering and Rigorous Validation to Create Comprehensive Synthetic Datasets for AI Training in Healthcare
Deep Learning Characterizes Depression and Suicidal Ideation from Eye Movements
Multimodal Large Language Models for Medicine: A Comprehensive Survey
Artificial Intelligence for Personalized Prediction of Alzheimer's Disease Progression: A Survey of Methods, Data Challenges, and Future Directions
MDD-LLM: Towards Accuracy Large Language Models for Major Depressive Disorder Diagnosis
KoACD: The First Korean Adolescent Dataset for Cognitive Distortion Analysis