多模态推理推进对话式诊断AI
研究开发了多模态扩展的Articulate Medical Intelligence Explorer(multimodal AMIE),采用状态感知对话框架,在105次模拟远程会诊中与初级保健医生进行随机盲法探索性比较,由18名专科医生评估。结果显示multimodal AMIE在诊断准确性和对话质量(包括病史采集和共情)上均优于PCPs,在32个评估轴中的29个表现更优。
要点
- 多模态AMIE在模拟远程会诊中诊断准确性优于初级保健医生。
- 对话质量(病史采集、共情)也优于初级保健医生。
- 状态感知推理可桥接文本与视觉信息,辅助复杂多模态诊断。
核心结论
- 多模态AMIE在诊断准确性和对话质量上优于初级保健医生,验证了状态感知推理在多模态诊断中的有效性。
研究设计与样本
- 随机、盲法探索性研究,纳入105次模拟远程会诊,包含皮肤照片、心电图和临床文档。
- 由18名专科医生评估多模态AMIE与初级保健医生的表现。
主要发现
- 多模态AMIE在32个评估轴中的29个表现更优,包括9个多模态推理指标中的7个。
- 多模态AMIE在诊断准确性、病史采集和共情方面均优于初级保健医生。
临床怎么用
- 多模态AMIE有望在复杂多模态诊断场景中辅助临床医生。
局限性
- 研究为模拟远程会诊的探索性研究,非真实临床环境。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Real-world clinical practice is inherently multimodal, relying on the synthesis of patient history with visual information such as medical imagery and clinical documents. Although large language models (LLMs) have shown promise in diagnostic dialogue, their evaluation has been largely restricted to text-only interactions, failing to capture the complexity of modern remote care delivery. Here we introduce a multimodal extension of the Articulate Medical Intelligence Explorer (multimodal AMIE), capable of gathering, interpreting and reasoning about multimodal data within a diagnostic conversation. To achieve this, we developed a state-aware dialogue framework that dynamically guides history-taking based on diagnostic uncertainty and evolving patient states, emulating the structured reasoning of experienced clinicians. We evaluated this updated, state-aware version of multimodal AMIE against primary care physicians (PCPs) in a randomized, blinded exploratory study comprising 105 simulated telehealth consultations, which included dermatology photographs, electrocardiograms and clinical documents. As assessed by 18 specialist physicians, multimodal AMIE outperformed PCPs not only in diagnostic accuracy but also in conversation quality, including history-taking and empathy. Specifically, multimodal AMIE demonstrated superior performance on 29 of 32 evaluation axes, including seven of nine metrics that assess multimodal reasoning. These results validate the efficacy of state-aware reasoning in bridging the gap between text and visual information and demonstrate the potential for artificial intelligence (AI) systems to augment clinicians in complex, multimodal diagnostic settings.
引用
- Saab K, Park C, Strother T, et al. Advancing conversational diagnostic AI with multimodal reasoning. Nat Med. 2026. PMID: 42135531. doi:10.1038/s41591-026-04371-0
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。