文献速递医学人工智能与信息学医学教育与科研方法

大型语言模型在患者互动任务中临床应用的评估框架

An evaluation framework for clinical use of large language models in patient interaction tasks.
PubMed 文献 · Nat Med · 2025-01-02

本文提出CRAFT-MD(医学测试对话推理评估框架),用模拟AI代理与LLM在受控环境中进行自然对话,评估GPT-4、GPT-3.5、Mistral和LLaMA-2-7b在12个专科的诊断能力。实验揭示当前LLM在临床对话推理、病史采集和诊断准确性方面的局限,GPT-4V的多模态对话与视觉评估能力同样存在这些局限。作者基于实证提出未来临床LLM评估建议,强调真实医患对话、全面病史采集、开放式提问及自动化与专家评估结合。

要点

  • CRAFT-MD用模拟AI代理与LLM自然对话,评估临床对话推理、病史采集和诊断准确性。
  • GPT-4、GPT-3.5、Mistral和LLaMA-2-7b在12个专科的评估中均显示临床对话推理与诊断准确性存在局限。
  • GPT-4V的多模态对话与视觉评估能力同样存在这些局限。

核心结论

  • 当前LLM在临床对话推理、病史采集和诊断准确性方面存在关键局限,GPT-4V的多模态对话与视觉评估能力同样存在这些局限。
  • 作者提出未来临床LLM评估应基于真实医患对话、全面病史采集、开放式提问以及自动化与专家评估相结合。

研究设计与样本

  • 本文提出CRAFT-MD(医学测试对话推理评估框架),用模拟AI代理在受控环境中与LLM进行自然对话,而非依赖结构化医学考试。
  • 应用CRAFT-MD评估GPT-4、GPT-3.5、Mistral和LLaMA-2-7b在12个医学专科的诊断能力。

主要发现

  • 实验揭示当前LLM在临床对话推理、病史采集和诊断准确性方面的关键局限。
  • 这些局限在分析GPT-4V的多模态对话与视觉评估能力时同样存在。

临床怎么用

  • 作者建议未来临床LLM评估应强调真实医患对话、全面病史采集、开放式提问以及自动化与专家评估相结合。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

The integration of large language models (LLMs) into clinical diagnostics has the potential to transform doctor-patient interactions. However, the readiness of these models for real-world clinical application remains inadequately tested. This paper introduces the Conversational Reasoning Assessment Framework for Testing in Medicine (CRAFT-MD) approach for evaluating clinical LLMs. Unlike traditional methods that rely on structured medical examinations, CRAFT-MD focuses on natural dialogues, using simulated artificial intelligence agents to interact with LLMs in a controlled environment. We applied CRAFT-MD to assess the diagnostic capabilities of GPT-4, GPT-3.5, Mistral and LLaMA-2-7b across 12 medical specialties. Our experiments revealed critical insights into the limitations of current LLMs in terms of clinical conversational reasoning, history-taking and diagnostic accuracy. These limitations also persisted when analyzing multimodal conversational and visual assessment capabilities of GPT-4V. We propose a comprehensive set of recommendations for future evaluations of clinical LLMs based on our empirical findings. These recommendations emphasize realistic doctor-patient conversations, comprehensive history-taking, open-ended questioning and using a combination of automated and expert evaluations. The introduction of CRAFT-MD marks an advancement in testing of clinical LLMs, aiming to ensure that these models augment medical practice effectively and ethically.

来源:Nat Med,查看原文。版权归原作者所有。

引用

  1. Johri S, Jeong J, Tran BA, et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat Med. 2025. PMID: 39747685. doi:10.1038/s41591-024-03328-5

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。