大语言模型生成的围术期神经认知障碍推荐与已发表指南的比较:一项横断面网络分析
该横断面网络分析于2024年6月在五个国家六个地点在线进行,将ChatGPT-4和Gemini生成的围术期神经认知障碍(PND)管理推荐与已发表指南比较。主要结局为总不一致评分(TDS)和医学人工智能质量评估(QAMAI)。结果显示两者TDS和QAMAI相似,AI推荐与指南总体对齐良好但无完全一致,且缺乏引用来源。
要点
- ChatGPT-4与Gemini生成的PND推荐在TDS和QAMAI上无显著差异。
- AI推荐与已发表指南对齐良好,但未达到完全一致。
- 缺乏引用来源是AI生成推荐的主要局限,临床应用前需验证。
核心结论
- 大语言模型可生成与已发表指南密切对齐的PND推荐,但临床应用前需进一步验证并整合临床医生反馈。
研究设计与样本
- 2024年6月进行的为期48小时的在线横断面网络分析,在五个国家(瑞士、比利时、土耳其、加拿大、美国东海岸和西海岸)的六个地点生成AI推荐。
- 使用英文提示「a table of a bundle of care for perioperative neurocognitive disorders」输入ChatGPT-4和Gemini生成表格,由独立评审员评估。
- 主要结局为总不一致评分(TDS)和医学人工智能质量评估(QAMAI),比较AI生成推荐与已发表指南。
主要发现
- 研究共生成14个表格。
- ChatGPT-4与Gemini的TDS评分分别为2 [1-3]和2 [2-3],P=0.636。
- ChatGPT-4与Gemini的QAMAI评分分别为4 [4-4]和4 [3-4],P=0.424。
- AI生成推荐与已发表指南对齐良好,其中ChatGPT-4生成的推荐对齐度最高。
- 未达到与指南的完全一致,缺乏引用来源是已知局限。
临床怎么用
- 大语言模型生成的PND推荐与指南对齐良好,但尚不能直接用于临床决策,需结合临床医生反馈和进一步验证。
局限性
- AI生成推荐缺乏引用来源。
- 未达到与指南的完全一致。
- 研究为在线横断面分析,仅持续48小时,样本为14个表格。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Perioperative neurocognitive disorders (PNDs) are common complications after surgery and anaesthesia, particularly in older adults, leading to increased morbidity, mortality, and healthcare costs. Therefore, major medical societies have developed recommendations for the prevention and treatment of PNDs. Our study evaluated the reliability of large language models, specifically ChatGPT-4 and Gemini, in generating recommendations for PND management and comparing them with published guidelines. METHODS: We conducted an online cross-sectional web-based analysis over 48 h in June 2024. Artificial intelligence (AI)-generated recommendations were produced in six different locations across five countries (Switzerland, Belgium, Turkey, Canada, and the East and West Coasts of the USA). The English prompt 'a table of a bundle of care for perioperative neurocognitive disorders' was entered into ChatGPT-4 and Gemini, generating tables evaluated by independent reviewers. The primary outcomes were the Total Disagreement Score (TDS) and Quality Assessment of Medical Artificial Intelligence (QAMAI), which compared AI-generated recommendations with published guidelines. RESULTS: The study generated 14 tables, with TDS and QAMAI scores showing similar results for ChatGPT-4 and Gemini (2 [1-3] vs 2 [2-3], P=0.636 and 4 [4-4] vs 4 [3-4], P=0.424, respectively). AI-generated recommendations aligned well with published guidelines, with the highest alignment observed in ChatGPT-4-generated recommendations. No complete agreement with guidelines was achieved, and lack of cited sources was a noted limitation. CONCLUSIONS: Large language models can generate perioperative neurocognitive disorder recommendations that align closely with published guidelines. However, further validation and integration of clinician feedback are required before clinical application.
引用
- Saxena S, Barreto Chang OL, Suppan M, et al. A comparison of large language model-generated and published perioperative neurocognitive disorder recommendations: a cross-sectional web-based analysis. Br J Anaesth. 2025. PMID: 39922789. doi:10.1016/j.bja.2025.01.001
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。