大语言模型在文献筛选中的表现
本研究为前瞻性诊断研究,使用日本脓毒症与脓毒性休克管理临床实践指南制定中5个临床问题的题录摘要筛选数据,比较GPT-4 Turbo辅助筛选与传统方法,以传统方法全文筛选结果为参考标准。主要分析中LLM辅助筛选的汇总敏感度为0.75(95%CI 0.43-0.92),特异度为0.99(95%CI 0.99-0.99)。事后修改命令提示后敏感度提高至0.91(95%CI 0.77-0.97),特异度为0.98(95%CI 0.96-0.99)。LLM辅助筛选处理100项研究耗时1.3分钟,传统方法为17.2分钟,平均差-15.25分钟(95%CI -17.70至-12.79分钟)。作者认为该方法可能提高系统评价效率并减少工作量。
要点
- 在5个临床问题的题录摘要筛选中,GPT-4 Turbo辅助筛选的汇总敏感度为0.75(95%CI 0.43-0.92),特异度为0.99(95%CI 0.99-0.99)。
- 事后修改命令提示后,汇总敏感度提高至0.91(95%CI 0.77-0.97),特异度为0.98(95%CI 0.96-0.99)。
- LLM辅助筛选处理100项研究耗时1.3分钟,传统方法为17.2分钟,平均差-15.25分钟(95%CI -17.70至-12.79分钟)。
- 传统方法筛选出的文献数:临床问题1为5634篇中8篇,临床问题2为3418篇中4篇,临床问题3为1038篇中4篇,临床问题4为4326篇中17篇,临床问题5为2253篇中8篇。
- 作者认为该新方法可能提高系统评价效率并减少工作量。
适用人群与场景
- 日本脓毒症与脓毒性休克管理临床实践指南制定中5个临床问题的题录摘要筛选数据。
研究设计与关键结果
- 研究类型:前瞻性诊断研究;数据来源:日本脓毒症与脓毒性休克管理临床实践指南制定中5个临床问题的题录摘要筛选过程;研究时间:2024年1月7日至15日。
- 传统方法筛选结果:临床问题1为5634篇中8篇,临床问题2为3418篇中4篇,临床问题3为1038篇中4篇,临床问题4为4326篇中17篇,临床问题5为2253篇中8篇。
- 主要分析:LLM辅助筛选汇总敏感度0.75(95%CI 0.43-0.92),汇总特异度0.99(95%CI 0.99-0.99)。
- 事后修改命令提示后:汇总敏感度0.91(95%CI 0.77-0.97),汇总特异度0.98(95%CI 0.96-0.99)。
- 筛选耗时:LLM辅助筛选处理100项研究1.3分钟 vs 传统方法17.2分钟,平均差-15.25分钟(95%CI -17.70至-12.79分钟)。
证据类型、等级与局限
- 证据类型:前瞻性诊断研究。
- 局限:原文未报告局限性。
对床旁实践的改变
- 作者认为,LLM辅助题录摘要筛选可能提高系统评价效率并减少工作量。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
IMPORTANCE: Large language models (LLMs) are promising as tools for citation screening in systematic reviews. However, their applicability has not yet been determined.
OBJECTIVE: To evaluate the accuracy and efficiency of an LLM in title and abstract literature screening.
DESIGN, SETTING, AND PARTICIPANTS: This prospective diagnostic study used the data from the title and abstract screening process for 5 clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock. The LLM decided to include or exclude citations based on the inclusion and exclusion criteria in terms of patient, population, problem; intervention; comparison; and study design of the selected CQ and was compared with the conventional method for title and abstract screening. This study was conducted from January 7 to 15, 2024.
EXPOSURES: LLM (GPT-4 Turbo)-assisted citation screening or the conventional method.
MAIN OUTCOMES AND MEASURES: The sensitivity and specificity of the LLM-assisted screening process was calculated, and the full-text screening result using the conventional method was set as the reference standard in the primary analysis. Pooled sensitivity and specificity were also estimated, and screening times of the 2 methods were compared.
RESULTS: In the conventional citation screening process, 8 of 5634 publications in CQ 1, 4 of 3418 in CQ 2, 4 of 1038 in CQ 3, 17 of 4326 in CQ 4, and 8 of 2253 in CQ 5 were selected. In the primary analysis of 5 CQs, LLM-assisted citation screening demonstrated an integrated sensitivity of 0.75 (95% CI, 0.43 to 0.92) and specificity of 0.99 (95% CI, 0.99 to 0.99). Post hoc modifications to the command prompt improved the integrated sensitivity to 0.91 (95% CI, 0.77 to 0.97) without substantially compromising specificity (0.98 [95% CI, 0.96 to 0.99]). Additionally, LLM-assisted screening was associated with reduced time for processing 100 studies (1.3 minutes vs 17.2 minutes for conventional screening methods; mean difference, -15.25 minutes [95% CI, -17.70 to -12.79 minutes]).
CONCLUSIONS AND RELEVANCE: In this prospective diagnostic study investigating the performance of LLM-assisted citation screening, the model demonstrated acceptable sensitivity and reasonably high specificity with reduced processing time. This novel method could potentially enhance efficiency and reduce workload in systematic reviews.
参考来源
- Oami T, Okada Y, Nakada TA, et al. Performance of a Large Language Model in Screening Citations. JAMA Netw Open. 2024. PMID: 38976267. doi:10.1001/jamanetworkopen.2024.20496
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。