大型语言模型在临床决策中的局限性的评估与缓解
研究基于重症监护医学信息数据库构建了涵盖2400例真实患者病例和四种常见腹部病变的数据集及模拟真实临床环境的框架,评估当前最先进大型语言模型在临床决策中的表现。结果显示这些模型在所有病变上的诊断准确率均显著低于医生,不遵循诊断或治疗指南,无法解读实验室结果,且难以整合到现有工作流程中。分析表明大型语言模型目前尚不适合自主临床决策。
要点
- 当前最先进大型语言模型在所有病变上的诊断表现均显著差于医生。
- 模型不遵循诊断或治疗指南,且无法解读实验室结果。
- 模型难以整合到现有临床工作流程,常不遵循指令且对信息数量和顺序敏感。
核心结论
- 当前大型语言模型尚不适合自主临床决策,并提供了数据集与框架以指导未来研究。
研究设计与样本
- 基于重症监护医学信息数据库构建了涵盖2400例真实患者病例和四种常见腹部病变的精选数据集及模拟真实临床环境的框架。
主要发现
- 当前最先进大型语言模型在所有病变上的诊断准确率均显著低于医生。
- 大型语言模型不遵循诊断指南也不遵循治疗指南。
- 大型语言模型无法解读实验室结果。
- 大型语言模型常不遵循指令,且对信息的数量和顺序敏感,难以整合到现有工作流程中。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.
引用
- Hager P, Jungmann F, Holland R, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat Med. 2024. PMID: 38965432. doi:10.1038/s41591-024-03097-1
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。