文献速递医学人工智能与信息学医学教育与科研方法

大语言模型在部分系统综述任务中表现良好但需谨慎实施:一项系统综述

Large language models show promising performance for some systematic review tasks but call for cautious implementation: a systematic review.
PubMed 文献 · J Clin Epidemiol · 2026-03-12

该研究系统检索了截至2025年1月14日的文献,纳入63项评估大语言模型在系统综述各步骤中性能的研究。结果显示,大语言模型在标题摘要筛选和全文筛选中与人类评审员具有良好一致性,较新模型表现更优,但在数据提取和偏倚风险评估等复杂任务中性能变异较大。结论认为大语言模型可辅助系统综述但不应替代人工判断。

要点

  • 大语言模型在标题摘要筛选和全文筛选中与人类评审员一致性较高。
  • 较新代大语言模型(GPT-4之后)在筛查任务中表现优于早期模型。
  • 复杂任务如数据提取和偏倚风险评估的性能变异大,需人工监督。

核心结论

  • 大语言模型在系统综述的筛查等重复性步骤中展现出应用前景,但成功整合需要适当保障措施和谨慎实施。

研究设计与样本

  • 系统综述,检索PubMed、Embase、Cochrane Library和预印本平台至2025年1月14日,纳入63项评估大语言模型在系统综述任何步骤中性能的研究。

主要发现

  • 标题摘要筛选中,大语言模型与人类评审员的总体中位阳性一致率(PPA)为0.92(IQR 0.69-0.98),中位阴性一致率(NPA)为0.89(IQR 0.72-0.95)。
  • 全文筛选中,总体中位PPA为0.93(IQR 0.87-1.00),中位NPA为0.92(IQR 0.78-0.97)。
  • 数据提取任务的总体准确率范围为0.36至1.00,中位准确率为0.95(IQR 0.91-0.97,n=11)。
  • 偏倚风险评估任务的准确率范围为0.44至0.90,中位准确率为0.62(IQR 0.53-0.76,n=6)。

关键数值与推荐

  • 共纳入63项研究,其中52项报告了性能指标,涵盖148项大语言模型性能评估。
  • 大多数评估针对GPT模型(n=114,77%)。
  • 最常评估的任务为标题摘要筛选(n=78,53%)、数据提取(n=23,16%)和全文筛选(n=20,14%)。

临床怎么用

  • 大语言模型可作为系统综述的辅助工具,用于筛选等重复性任务以提高效率。
  • 在数据提取和偏倚风险评估等复杂任务中,需谨慎使用并加强人工监督。
  • 应用时应建立适当保障措施、透明报告和人工审核机制。

局限性

  • 其他任务的性能指标变异较大,妨碍了完整的定量合成。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: With the exponential growth of biomedical literature, the challenge of conducting systematic reviews is becoming increasingly burdensome. We aimed to evaluate the performance of large language models (LLMs) in the automation of some or all steps of systematic reviews and meta-analyses. STUDY DESIGN AND SETTING: In this systematic review, we searched PubMed, Embase, the Cochrane Library and preprint platforms up to January 14, 2025. We included any studies assessing the performance of LLMs (eg, generative pre-trained transformer [GPT], Claude, Mistral) in any step of the systematic review process. Pairs of reviewers independently extracted data and assessed risk of bias. We conducted analyses using median (interquartile range [IQR]) for positive (PPA) and negative percent agreements (NPA), respectively, analogous to sensitivity and specificity, between LLMs and human reviewers. RESULTS: From 3889 unique references, we included 63 studies of which 52 reporting performance metrics for a total of 148 LLM performance assessments. Most assessments concerned GPT models (n = 114, 77%). The most frequently evaluated tasks were title and abstract screening (n = 78, 53%), data extraction (n = 23, 16%), and full-text screening (n = 20, 14%). For title and abstract screening, overall median PPA was 0.92 (IQR 0.69-0.98) and median NPA was 0.89 (0.72-0.95). For full-text screening, the overall median PPA was 0.93 (0.87-1.00) and median NPA was 0.92 (0.78-0.97). Late-generation LLMs released after GPT-4 seemed to provide higher performance than earlier models. For other tasks, authors reported overall good performances, but variability of performance metrics precluded complete quantitative synthesis. Global accuracy for data extraction tasks ranged from 0.36 to 1.00, with a median accuracy of 0.95 (IQR 0.91-0.97, n = 11). For the "risk of bias assessment" task, accuracy ranged from 0.44 to 0.90 (median = 0.62, IQR 0.53-0.76, n = 6). CONCLUSIONS: The performance of LLMs, particularly newer generations, shows promise in automating some repetitive steps of systematic reviews such as screening. However, their successful integration will require appropriate safeguards and careful implementation. PLAIN LANGUAGE SUMMARY: Systematic reviews are one of the most reliable ways to answer medical and public health questions. They bring together all available studies on a topic and help clinicians and policymakers make informed decisions. However, producing a high-quality systematic review takes a lot of time and effort. Whole teams of researchers spend months screening thousands of articles, extracting data, and double-checking results. With little more than a million of new publications every year, keeping reviews up to date is becoming increasingly difficult. LLMs, such as ChatGPT, may help reduce this workload. These tools can read and summarize text and might assist with repetitive tasks like selecting relevant studies or extracting information from articles. But it is still unclear how reliable these tools are for research purposes. This is the first systematic review to assess LLMs' performance to facilitate systematic reviews. We sought to review all studies that tested LLMs in the different steps of systematic reviews and found 63 studies evaluating how well these tools performed compared with human reviewers. Overall, LLMs showed good agreement with humans for tasks such as screening titles and abstracts, and full-text articles. Newer models seemed to perform better than older ones. However, performance was more variable for complex tasks that require interpretation, such as extracting detailed data or assessing methodological quality. Our findings suggest that LLMs could help researchers work faster and make systematic reviews more efficient. However, they are not ready to replace human judgment. These tools can make mistakes, produce inconsistent results, or generate inaccurate information if not carefully supervised. In practice, LLMs should be used as assistants rather than substitutes. With proper safeguards, transparent reporting, and human oversight, they may become valuable tools to support evidence-based healthcare and help keep research up to date.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Laignelot F, Martin GL, Ossman M, et al. Large language models show promising performance for some systematic review tasks but call for cautious implementation: a systematic review. J Clin Epidemiol. 2026. PMID: 41831731. doi:10.1016/j.jclinepi.2026.112221

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。