大型语言模型在患者互动任务中临床应用的评估框架
本文提出CRAFT-MD(医学测试对话推理评估框架),用模拟AI代理与LLM在受控环境中进行自然对话,评估GPT-4、GPT-3.5、Mistral和LLaMA-2-7b在12个专科的诊断能力。实验揭示当前LLM在临床对话推理、病史采集和诊断准确性方面的局限,GPT-4V的多模态对话与视觉评估能力同样存在这些局限。作者基于实证提出未来临床LLM评估建议,强调真实医患对话、全面病史采集、开放式提问及自动化与专家评估结合。
要点
- CRAFT-MD用模拟AI代理与LLM自然对话,评估临床对话推理、病史采集和诊断准确性。
- GPT-4、GPT-3.5、Mistral和LLaMA-2-7b在12个专科的评估中均显示临床对话推理与诊断准确性存在局限。
- GPT-4V的多模态对话与视觉评估能力同样存在这些局限。
核心结论
- 当前LLM在临床对话推理、病史采集和诊断准确性方面存在关键局限,GPT-4V的多模态对话与视觉评估能力同样存在这些局限。
- 作者提出未来临床LLM评估应基于真实医患对话、全面病史采集、开放式提问以及自动化与专家评估相结合。
研究设计与样本
- 本文提出CRAFT-MD(医学测试对话推理评估框架),用模拟AI代理在受控环境中与LLM进行自然对话,而非依赖结构化医学考试。
- 应用CRAFT-MD评估GPT-4、GPT-3.5、Mistral和LLaMA-2-7b在12个医学专科的诊断能力。
主要发现
- 实验揭示当前LLM在临床对话推理、病史采集和诊断准确性方面的关键局限。
- 这些局限在分析GPT-4V的多模态对话与视觉评估能力时同样存在。
临床怎么用
- 作者建议未来临床LLM评估应强调真实医患对话、全面病史采集、开放式提问以及自动化与专家评估相结合。
局限性
- 原文未报告局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
The integration of large language models (LLMs) into clinical diagnostics has the potential to transform doctor-patient interactions. However, the readiness of these models for real-world clinical application remains inadequately tested. This paper introduces the Conversational Reasoning Assessment Framework for Testing in Medicine (CRAFT-MD) approach for evaluating clinical LLMs. Unlike traditional methods that rely on structured medical examinations, CRAFT-MD focuses on natural dialogues, using simulated artificial intelligence agents to interact with LLMs in a controlled environment. We applied CRAFT-MD to assess the diagnostic capabilities of GPT-4, GPT-3.5, Mistral and LLaMA-2-7b across 12 medical specialties. Our experiments revealed critical insights into the limitations of current LLMs in terms of clinical conversational reasoning, history-taking and diagnostic accuracy. These limitations also persisted when analyzing multimodal conversational and visual assessment capabilities of GPT-4V. We propose a comprehensive set of recommendations for future evaluations of clinical LLMs based on our empirical findings. These recommendations emphasize realistic doctor-patient conversations, comprehensive history-taking, open-ended questioning and using a combination of automated and expert evaluations. The introduction of CRAFT-MD marks an advancement in testing of clinical LLMs, aiming to ensure that these models augment medical practice effectively and ethically.
引用
- Johri S, Jeong J, Tran BA, et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med. 2025. PMID: 39747685. doi:10.1038/s41591-024-03328-5
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。