大语言模型对多模态重症医学考试题给出准确但可能不安全的答案
本研究为观察性研究,使用重症医学学会题库中 183 道附带临床图像的多选题,在模拟重症医学考试环境下评估 ChatGPT-4o 的表现,由 14 名临床评审者(医生、高级实践提供者和药师)评审。ChatGPT-4o 总正确率 74.9%,高于临床医生汇总的 71.1%(p=0.03);问题理解正确率 87.4%,但图像解读 61.7%、推理 68.3%、支持信息 66.1% 较低。其在肺部疾病(91.7%)、外科与创伤(87.5%)、神经系统疾病(81.8%)表现较好,在重症超声(51.1%)表现较差。33.3% 的回答有潜在临床危害,常因图像解读错误和治疗建议不当。
要点
- ChatGPT-4o 答对 74.9% 的重症医学考试题,高于 14 名临床评审者汇总的 71.1%(p=0.03)。
- ChatGPT-4o 图像解读正确率仅 61.7%,推理 68.3%,支持信息 66.1%,均低于问题理解 87.4%。
- 33.3% 的 ChatGPT-4o 回答有潜在临床危害,常因图像解读错误和治疗建议不当。
- ChatGPT-4o 在重症超声题目正确率 51.1%,在肺部疾病 91.7%、外科与创伤 87.5%、神经系统疾病 81.8%。
- 作者认为 ChatGPT-4o 推理和图像分析缺陷可能导致高风险临床决策或临床教育中的有害结论。
适用人群与场景
- 评估对象为重症医学学会题库中 183 道附带临床图像的多选题,涵盖多个重症医学领域和影像模态。
- 评审者为 14 名临床评审者(医生、高级实践提供者和药师)。
研究设计与关键结果
- 观察性研究,使用重症医学学会题库 183 道附带临床图像的多选题,在模拟重症医学考试环境下评估 ChatGPT-4o,由 14 名临床评审者评审。
- ChatGPT-4o 总正确率 74.9%,高于临床医生汇总的 71.1%(p=0.03)。
- ChatGPT-4o 问题理解正确率 87.4%,图像解读 61.7%,推理 68.3%,支持信息 66.1%。
- ChatGPT-4o 在肺部疾病正确率 91.7%,外科与创伤 87.5%,神经系统疾病 81.8%,重症超声 51.1%。
- 33.3% 的 ChatGPT-4o 回答有潜在临床危害,常因图像解读错误和治疗建议不当。
证据类型、等级与局限
- 证据类型:观察性研究。
- 局限:原文未报告局限性。
对床旁实践的改变
- 作者认为 ChatGPT-4o 推理和图像分析缺陷可能导致高风险临床决策或临床教育中的有害结论。
- 原文未据此给出推荐。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
OBJECTIVES: To evaluate the performance of Chat Generative Pre-Trained Transformer-4 Omni (ChatGPT-4o) in answering multimodal critical care board review questions, with a focus on accuracy, image interpretation, reasoning quality, and potential for harm.
DESIGN: Observational study using a validated item bank of multiple-choice questions accompanied by clinical images, analyzed through a custom ChatGPT-4o profile built using a validated framework.
SETTING: Simulated environment mimicking critical care board examination conditions, with artificial intelligence responses reviewed by a panel of experienced critical care clinicians.
SUBJECTS: One hundred eighty-three board-style questions from the Society of Critical Care Medicine item bank, representing a range of critical care domains and imaging modalities. ChatGPT-4o was evaluated on its responses, which were assessed by 14 clinical reviewers (physicians, advanced practice providers, and pharmacists).
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: ChatGPT-4o answered 74.9% of questions correctly, higher than pooled clinician responses (71.1%; p = 0.03). It showed strengths in question comprehension (87.4% correct) but lower performance in image interpretation (61.7%), reasoning (68.3%), and supporting information (66.1%). ChatGPT-4o excelled in pulmonary disease (91.7%), surgery and trauma (87.5%), and neurologic disorders (81.8%), and underperformed in critical care ultrasound (51.1%). Notably, 33.3% of its responses were associated with potential for clinical harm, often due to incorrect image interpretation and treatment recommendations.
CONCLUSIONS: ChatGPT-4o demonstrates performance slightly above pooled clinician benchmarks on critical care board-style questions but has substantial limitations in multimodal question interpretation. Despite its high comprehension, deficiencies in ChatGPT-4o's reasoning and image analysis may lead to harmful clinical conclusions in high-stakes clinical decision-making or clinical education.
参考来源
- Sethi I, Khan S, Lyons PG, et al. Large Language Models Provide Accurate but Potentially Unsafe Answers to Multimodal Critical Care Medicine Board Review Questions. Crit Care Med. 2026. PMID: 42307255. doi:10.1097/CCM.0000000000007228
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。