ChatGPT、ChatGPT-4与Google Bard在高级烧伤生命支持考试中的表现比较分析
该研究于2023年7月18日使用最新版ABLS考试的50道多选题比较ChatGPT-3.5、ChatGPT-4和Google Bard的表现,及格线为80%。ChatGPT-3.5得分86%(43/50),ChatGPT-4得分90%(45/50),Bard得分70%(35/50)。ChatGPT-4显著优于Bard(P = .012),ChatGPT-3.5与ChatGPT-4及Bard间无显著差异。
要点
- ChatGPT-3.5和ChatGPT-4在ABLS考试中准确率分别为86%和90%,均超过80%及格线。
- ChatGPT-4得分显著高于Google Bard(P = .012),Bard得分70%未达及格线。
- ChatGPT-3.5与ChatGPT-4及Bard间差异未达统计学显著(P = .538和P = .054)。
核心结论
- ChatGPT-3.5和ChatGPT-4在ABLS考试中表现出高准确率并优于Google Bard。
- 大型语言模型在急诊烧伤和创伤护理中的潜在多种应用需要适当的监管,且很可能应作为补充人类认知的工具。
研究设计与样本
- 研究于2023年7月18日使用最新版ABLS考试的50道多选题(每题5个选项,及格线80%)对ChatGPT-3.5、ChatGPT-4和Google Bard进行查询比较。
主要发现
- ChatGPT-3.5得分86%(43/50),ChatGPT-4得分90%(45/50),Bard得分70%(35/50)。
- ChatGPT-3.5与ChatGPT-4间无差异(P = .538),ChatGPT-3.5与Bard间无差异(P = .054),尽管P值处于临界。
- ChatGPT-4显著优于Bard(P = .012)。
- 50道题中78%(n = 39)为直接问题,12%(n = 11)以临床情景呈现;3种模型错误答案率按问题类型无差异。
临床怎么用
- 原文结论指出大型语言模型在急诊烧伤和创伤护理中的潜在多种应用需要适当的监管,且很可能应作为补充人类认知的工具。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Artificial intelligence and large language models (LLMs) have recently gained attention as promising tools in various healthcare domains, offering potential benefits in clinical decision-making, medical education, and research. The Advanced Burn Life Support (ABLS) program is a didactic initiative endorsed by the American Burn Association, aiming to provide knowledge on the immediate care of severely burn patients. The aim of the study was to compare the performance of three LLMs (ChatGPT-3.5, ChatGPT-4, and Google Bard) on the ABLS exam. The ABLS exam consists of 50 questions with 5 multiple-choice answers. The passing threshold is 80% of correct answers. The 3 LLMs were queried with the 50 questions included in the latest version of the ABLS exam, on July 18th, 2023. ChatGPT-3.5 scored 86% (43 out of 50), ChatGPT-4 scored 90% (45 out of 50), and Bard scored 70% (35 out of 50). No difference was measured between ChatGPT-3.5 and ChatGPT-4 (P = .538) and between ChatGPT-3.5 and Bard (P = .054), despite the borderline P-value. ChatGPT-4 performed significantly better than Bard (P = .012). Out of the 50 questions, 78% (n = 39) were direct questions, while 12% (n = 11) were presented as clinical scenarios. No difference in the rate of wrong answers was found based on the type of question for the 3 LLMs. ChatGPT-3.5 and ChatGPT-4 demonstrated high accuracy at the ABLS exam and outperformed Google Bard. However, the potential multiple applications of LLMs in emergency burn and trauma care necessitate appropriate surveillance and most likely should represent a tool to complement human cognition.
引用
- Alessandri-Bonetti M, Liu HY, Donovan JM, et al. A Comparative Analysis of ChatGPT, ChatGPT-4, and Google Bard Performances at the Advanced Burn Life Support Exam. J Burn Care Res. 2024. PMID: 38833383. doi:10.1093/jbcr/irae044
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。