文献速递疼痛医学

人类与人工智能对比:评估ChatGPT在慢性疼痛系统综述与Meta分析中的表现

Human versus artificial intelligence: evaluating ChatGPT's performance in conducting published systematic reviews with meta-analysis in chronic pain research.
PubMed 文献 · Reg Anesth Pain Med · 2026-04-02

本研究以已发表的脊髓电刺激后情绪功能Meta分析为数据源,让ChatGPT-4o执行标题/摘要筛选、全文筛选和数据合并,并与人工操作对比。结果显示ChatGPT筛选准确率中等(标题摘要70.4%,全文68.4%),但数据合并与Meta分析计算准确率高(合并均数差及95%CI计算100%准确),提示AI可辅助系统综述流程但需人工监督。

要点

  • ChatGPT-4o在标题/摘要筛选中准确率70.4%,灵敏度54.9%,特异度80.1%。
  • 全文筛选准确率68.4%,灵敏度75.6%,特异度66.8%。
  • 数据合并与Meta分析计算表现良好,多数结局合并均数差及95%CI计算100%准确。

核心结论

  • ChatGPT在筛选和研究选择任务中表现出中等程度准确性,但在数据合并和Meta分析计算方面表现良好。
  • 研究结果强调AI有潜力辅助系统综述方法学,同时需要人工监督以确保研究流程的准确性和完整性。

研究设计与样本

  • 验证性研究,使用已发表的关于脊髓电刺激后情绪功能的Meta分析数据。
  • 由ChatGPT-4o执行标题/摘要筛选、全文研究选择和数据合并,并与人工执行的步骤(作为金标准)进行比较。

主要发现

  • 标题和摘要筛选:准确率70.4%,灵敏度54.9%,特异度80.1%。
  • 全文筛选:准确率68.4%,灵敏度75.6%,特异度66.8%。
  • ChatGPT成功为五个森林图合并数据,在计算合并均数差、95%CI和异质性估计(I2评分和tau-squared值)方面对大多数结局达到100%准确率,tau-squared值存在轻微差异(范围0.01-0.05)。
  • 森林图未显示显著差异。

关键数值与推荐

  • 标题/摘要筛选准确率70.4%、灵敏度54.9%、特异度80.1%。
  • 全文筛选准确率68.4%、灵敏度75.6%、特异度66.8%。
  • 合并均数差、95%CI和异质性估计(I2评分和tau-squared值)对大多数结局计算准确率100%,tau-squared值差异范围0.01-0.05。

临床怎么用

  • ChatGPT可用于辅助系统综述的筛选和数据合并流程,但筛选环节需人工复核以确保准确性。
  • 在Meta分析计算中ChatGPT表现良好,但仍需人工监督以确保研究完整性。

局限性

  • 本研究仅基于一个已发表的Meta分析数据,结果可能不适用于其他研究领域。
  • ChatGPT在筛选任务中准确率中等,灵敏度或特异度存在不足,需人工监督。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: Artificial intelligence (AI), particularly large-language models like Chat Generative Pre-Trained Transformer (ChatGPT), has demonstrated potential in streamlining research methodologies. Systematic reviews and meta-analyses, often considered the pinnacle of evidence-based medicine, are inherently time-intensive and demand meticulous planning, rigorous data extraction, thorough analysis, and careful synthesis. Despite promising applications of AI, its utility in conducting systematic reviews with meta-analysis remains unclear. This study evaluated ChatGPT's accuracy in conducting key tasks of a systematic review with meta-analysis. METHODS: This validation study used data from a published meta-analysis on emotional functioning after spinal cord stimulation. ChatGPT-4o performed title/abstract screening, full-text study selection, and data pooling for this systematic review with meta-analysis. Comparisons were made against human-executed steps, which were considered the gold standard. Outcomes of interest included accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for screening and full-text review tasks. We also assessed for discrepancies in pooled effect estimates and forest plot generation. RESULTS: For title and abstract screening, ChatGPT achieved an accuracy of 70.4%, sensitivity of 54.9%, and specificity of 80.1%. In the full-text screening phase, accuracy was 68.4%, sensitivity 75.6%, and specificity 66.8%. ChatGPT successfully pooled data for five forest plots, achieving 100% accuracy in calculating pooled mean differences, 95% CIs, and heterogeneity estimates (I2 score and tau-squared values) for most outcomes, with minor discrepancies in tau-squared values (range 0.01-0.05). Forest plots showed no significant discrepancies. CONCLUSION: ChatGPT demonstrates modest to moderate accuracy in screening and study selection tasks, but performs well in data pooling and meta-analytic calculations. These findings underscore the potential of AI to augment systematic review methodologies, while also emphasizing the need for human oversight to ensure accuracy and integrity in research workflows.

来源:Reg Anesth Pain Med,查看原文。版权归原作者所有。

引用

  1. Purewal A, Fautsch K, Klasova J, et al. Human versus artificial intelligence: evaluating ChatGPT's performance in conducting published systematic reviews with meta-analysis in chronic pain research. Reg Anesth Pain Med. 2026. PMID: 39956557. doi:10.1136/rapm-2024-106358

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。