大语言模型生成阑尾炎临床实践指南
本研究测试大语言模型(LLM)支持指南制定各阶段的能力,以SAGES阑尾炎手术管理指南为对照。ChatGPT-4、Google Gemini、Consensus和Perplexity能生成检索语法、进行数据分析并遵循GRADE方法,但无法独立完成系统检索、筛选、数据提取或偏倚风险评估。LLM生成指南的AGREE-S总分为119,SAGES指南为156。
要点
- LLM可辅助指南制定的部分步骤,如生成检索语法和数据分析。
- LLM尚不能独立完成系统文献检索、筛选、数据提取或偏倚风险评估。
- LLM生成指南的AGREE-S总分为119,低于SAGES指南的156。
核心结论
- LLM在指南制定的某些步骤中具有辅助潜力,可能减少时间和资源负担,但尚不能独立完成全部流程。
研究设计与样本
- 研究通过工程化提示词让LLM执行指南制定各任务,关键问题与PICO源自SAGES阑尾炎指南,于2024年2月21日查询ChatGPT-4、Google Gemini、Consensus和Perplexity,并进行定性评价和AGREE-S定量评估。
主要发现
- LLM能生成检索语法、进行数据分析并遵循GRADE方法和Evidence-to-Decision框架产生推荐。
- LLM无法独立进行系统文献检索,也不能可靠完成筛选、数据提取或偏倚风险评估。
- AGREE-S评估中,LLM生成指南总分为119,SAGES指南为156;24个域中有19个域两者得分相差在2分以内。
关键数值与推荐
- AGREE-S总分:LLM生成指南119分,SAGES指南156分。
- 24个AGREE-S域中,19个域两者得分差距不超过2分。
临床怎么用
- 可将LLM用于辅助生成检索语法、数据分析及遵循GRADE框架等步骤,但系统检索、筛选、数据提取和偏倚风险评估仍需人工完成。
局限性
- 测试仅于2024年2月21日进行,LLM能力可能随模型更新而变化;LLM无法独立完成系统检索和偏倚风险评估等关键步骤。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Clinical practice guidelines provide important evidence-based recommendations to optimize patient care, but their development is labor-intensive and time-consuming. Large language models have shown promise in supporting academic writing and the development of systematic reviews, but their ability to assist with guideline development has not been explored. In this study, we tested the capacity of LLMs to support each stage of guideline development, using the latest SAGES guideline on the surgical management of appendicitis as a comparison. METHODS: Prompts were engineered to trigger LLMs to perform each task of guideline development, using key questions and PICOs derived from the SAGES guideline. ChatGPT-4, Google Gemini, Consensus, and Perplexity were queried on February 21, 2024. LLM performance was evaluated qualitatively, with narrative descriptions of each task's output. The Appraisal of Guidelines for Research and Evaluation in Surgery (AGREE-S) instrument was used to quantitatively assess the quality of the LLM-derived guideline compared to the existing SAGES guideline. RESULTS: Popular LLMs were able to generate a search syntax, perform data analysis, and follow the GRADE approach and Evidence-to-Decision framework to produce guideline recommendations. These LLMs were unable to independently perform a systematic literature search or reliably perform screening, data extraction, or risk of bias assessment at the time of testing. AGREE-S appraisal produced a total score of 119 for the LLM-derived guideline and 156 for the SAGES guideline. In 19 of the 24 domains, the two guidelines scored within two points of each other. CONCLUSIONS: LLMs demonstrate potential to assist with certain steps of guideline development, which may reduce time and resource burden associated with these tasks. As new models are developed, the role for LLMs in guideline development will continue to evolve. Ongoing research and multidisciplinary collaboration are needed to support the safe and effective integration of LLMs in each step of guideline development.
引用
- Boyle A, Huo B, Sylla P, et al. Large language model-generated clinical practice guideline for appendicitis. Surg Endosc. 2025. PMID: 40251310. doi:10.1007/s00464-025-11723-3
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。