The field of medical large language models (LLMs) is rapidly evolving, with a growing focus on evaluating and improving their performance in real-world clinical settings. Recent studies have highlighted the importance of transparent and auditable reasoning processes, as well as the need for more comprehensive evaluation frameworks that go beyond single-turn question answering. Researchers are developing new benchmarks and evaluation tools, such as multi-turn dialogue datasets and automated pipelines, to assess the robustness and safety of medical LLMs. These efforts aim to address the limitations of current models, including their vulnerability to misleading context, authority influence, and other forms of perturbation. Noteworthy papers in this area include MedAgentAudit, which developed a comprehensive taxonomy of collaborative failure modes in medical multi-agent systems, and VivaBench, which introduced a multi-turn benchmark for evaluating sequential clinical reasoning in LLM agents. Additionally, EduDial constructed a large-scale multi-turn teacher-student dialogue corpus, and GAPS introduced a clinically grounded, automated benchmark for evaluating AI clinician systems.
Advancements in Evaluating and Improving Medical Large Language Models
Sources
MedAgentAudit: Diagnosing and Quantifying Collaborative Failure Modes in Medical Multi-Agent Systems
Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models
Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
SAGE: A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn AGent Evaluation
MedKGEval: A Knowledge Graph-Based Multi-Turn Evaluation Framework for Open-Ended Patient Interactions with Clinical LLMs
Shallow Robustness, Deep Vulnerabilities: Multi-Turn Evaluation of Medical LLMs
Using Medical Algorithms for Task-Oriented Dialogue in LLM-Based Medical Interviews
EduDial: Constructing a Large-scale Multi-turn Teacher-Student Dialogue Corpus
GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians