指南与共识消化、代谢与肾脏急症重症综合管理与镇静镇痛

评估大语言模型在急诊科提供临床建议的应用

Evaluating the use of large language models to provide clinical recommendations in the Emergency Department.
指南 / 共识 · Nat Commun · 2024-10-08
一句话结论在急诊科笔记的入院、影像检查和抗生素处方三项任务上,GPT-4-turbo和GPT-3.5-turbo的准确率分别比住院医师平均低8%和24%,作者认为部署为决策支持系统前性能需显著提升。

研究从急诊科随机选取10000次就诊记录,评估零样本条件下GPT-3.5-turbo和GPT-4-turbo在四种提示策略下对入院状态、影像学检查请求状态和抗生素处方状态三项任务给出临床建议的准确性。结果显示,GPT-4-turbo和GPT-3.5-turbo的准确率分别比住院医师平均低8%和24%。两种模型均倾向于过度谨慎,以高敏感度为代价牺牲特异度。作者认为,早期评估虽令人鼓舞,但大语言模型在作为临床建议决策支持系统部署前性能必须显著提升。

要点

  • 在10000次急诊就诊记录中,GPT-4-turbo和GPT-3.5-turbo对入院、影像检查请求和抗生素处方三项任务的准确率分别比住院医师平均低8%和24%。
  • 两种大语言模型均倾向于过度谨慎,推荐具有高敏感度但以牺牲特异度为代价。
  • 评估覆盖零样本条件下四种不同提示策略,GPT-4-turbo和GPT-3.5-turbo表现均差于住院医师。
  • 作者认为,大语言模型在作为临床建议决策支持系统部署前,性能必须显著提升。

适用人群与场景

  • 急诊科就诊患者的临床笔记,随机选取10000次急诊就诊记录

研究设计与关键结果

  • 研究类型为高统计效能评估研究,随机选取10000次急诊就诊,评估零样本、GPT-3.5-turbo和GPT-4-turbo在四种提示策略下对入院状态、影像学检查请求状态和抗生素处方状态三项任务给出临床建议的准确性。
  • GPT-4-turbo和GPT-3.5-turbo的准确率分别比住院医师平均低8%和24%。
  • 两种大语言模型均倾向于过度谨慎,推荐具有高敏感度但以牺牲特异度为代价。

证据类型、等级与局限

  • 证据类型:评估研究。
  • 局限:原文未报告局限性。

对床旁实践的改变

  • 作者认为,尽管大语言模型临床应用的早期评估结果令人鼓舞,但在作为临床建议决策支持系统部署前,其性能必须显著提升。

提炼说明

  • 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开

The release of GPT-4 and other large language models (LLMs) has the potential to transform healthcare. However, existing research evaluating LLM performance on real-world clinical notes is limited. Here, we conduct a highly-powered study to determine whether LLMs can provide clinical recommendations for three tasks (admission status, radiological investigation(s) request status, and antibiotic prescription status) using clinical notes from the Emergency Department. We randomly selected 10,000 Emergency Department visits to evaluate the accuracy of zero-shot, GPT-3.5-turbo- and GPT-4-turbo-generated clinical recommendations across four different prompting strategies. We found that both GPT-4-turbo and GPT-3.5-turbo performed poorly compared to a resident physician, with accuracy scores 8% and 24%, respectively, lower than physician on average. Both LLMs tended to be overly cautious in its recommendations, with high sensitivity at the cost of specificity. Our findings demonstrate that, while early evaluations of the clinical use of LLMs are promising, LLM performance must be significantly improved before their deployment as decision support systems for clinical recommendations and other complex tasks.

来源:Nat Commun,查看原文。版权归原作者所有。

参考来源

  1. Williams CYK, Miao BY, Kornblith AE, et al. Evaluating the use of large language models to provide clinical recommendations in the Emergency Department. Nat Commun. 2024. PMID: 39379357. doi:10.1038/s41467-024-52415-1

供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。