文献速递医学人工智能与信息学医学教育与科研方法

Cochrane(半)自动化综述方法评估:综述内自适应平台研究方案

Cochrane evaluation of (semi-)automated review methods: protocol for an adaptive platform study within reviews.
PubMed 文献 · J Clin Epidemiol · 2026-06-19

该方案由Cochrane CESAR项目成员制定,采用模拟临床试验的自适应平台研究设计,在系统综述流程中同时评估多个AI工具与纯人工流程的效果、效率与可用性。研究聚焦标题摘要筛选、全文筛选和数据提取三项任务,允许根据中期分析结果增减AI工具而无需重启研究,旨在为AI整合入证据合成工作流建立证据基础。

要点

  • 采用主方案自适应平台设计,可动态增减AI工具并持续评估。
  • 评估三项综述任务:标题摘要筛选、全文筛选、数据提取。
  • 评价指标包括准确性(灵敏度、特异度、精确率)、任务时间、稳定性、错误影响和可用性。

核心结论

  • 该方案提出一种新颖的方法学路径,通过验证完整工作流而非单一技术,为判定AI能否整合入证据合成工作流建立证据基础。
  • 自适应设计灵活,可被其他研究者采用,确保评估框架在新工具涌现时保持相关性。

研究设计与样本

  • Cochrane(半)自动化综述方法评估(CESAR)项目成员制定了综述内自适应平台研究设计,模拟临床平台试验。
  • 采用主方案同时评估多个AI工具(干预)与纯人工流程(对照),覆盖三项关键综述任务。
  • 自适应框架允许根据中期性能分析增减AI工具,无需重启研究。

主要发现

  • 研究将生成特定AI工具以半自动或全自动方式使用时的性能与可用性比较数据,对照标准人工投入。
  • 评估指标包括准确性(灵敏度、特异度、精确率)、效率(任务时间)、响应稳定性、错误影响和可用性,并遵循证据合成中负责任使用AI原则。

临床怎么用

  • 研究结果可为研究社区提供证据,判断何时及如何安全有效地在系统综述中使用AI。
  • 方案验证完整工作流而非单一技术,有助于指导AI工具在证据合成中的实际整合决策。

局限性

  • 原文指出快速演变的AI工具在证据合成实际工作流中的稳健评估方法仍不发达,本方案旨在应对该挑战。
  • 原文指出静态、一次性评估存在局限性,本方案通过自适应设计加以解决。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND AND OBJECTIVES: Artificial intelligence (AI) has the potential to improve the efficiency of evidence synthesis and reduce human error. However, robust methods for evaluating rapidly evolving AI tools within the practical workflows of evidence synthesis remain underdeveloped. This protocol describes a study design for assessing the effectiveness, efficiency, and usability of AI tools in comparison to traditional human-only workflows in the context of Cochrane systematic reviews. METHODS: Members of the Cochrane Evaluation of (Semi-)Automated Review Methods (CESAR) project developed an adaptive platform study-within-a-review design, modeled after clinical platform trials. This design employs a master protocol to concurrently evaluate multiple AI tools (interventions) against a standard human-only process (control) across 3 key review tasks: title and abstract screening, full-text screening, and data extraction. The adaptive framework allows for the addition or removal of AI tools based on interim performance analyses without necessitating a restart of the study. Performance will be assessed using metrics such as accuracy (sensitivity, specificity, precision), efficiency (time on task), response stability, impact of errors, and usability, in alignment with Responsible use of AI in evidence SynthEsis principles. RESULTS: The study will generate comparative data about the performance and usability of specific AI tools used in a semiautomated or fully automated manner relative to standard human effort. The protocol provides a flexible framework for the assessment of AI tools in evidence synthesis, addressing the limitations of static, one-time evaluations. CONCLUSION: This study protocol presents a novel methodological approach to addressing the challenges of evaluating AI tools for evidence syntheses. By validating entire workflows rather than individual technologies, the findings will establish an evidence base for determining the viability of integrating AI into evidence synthesis workflows. The adaptive design of this study is flexible and can be adopted by other investigators, ensuring that the evaluation framework remains relevant as new tools emerge. PLAIN LANGUAGE SUMMARY: Doctors and researchers rely on systematic reviews, which are thorough summaries of all available research on a health topic, to guide decisions about patient care. However, creating these reviews is a slow and demanding process, often taking more than a year to finish. AI tools could help speed up this work and reduce human errors, but there are currently no reliable ways to test how well these tools perform in real-world settings. This article describes the design of a study that will rigorously test how well AI tools perform when used in actual systematic review workflows, specifically within Cochrane Reviews. The study will compare AI-assisted methods with the traditional approach, where 2 trained researchers independently complete each step. It will look at 3 main tasks: choosing which studies might be relevant based on their titles and abstracts, reading the full-text publication to confirm which studies should be included, and extracting important information from those studies. A key strength of this study is its flexible design. Instead of testing just 1 AI tool at a single point in time, the study allows researchers to add or remove AI tools as new ones become available, similar to how some modern drug trials are run. This approach helps the study keep up with the fast pace of AI development. Researchers will assess the AI tools based on their accuracy, the time they save, how consistent their results are, and how easy they are to use. The ultimate goal of this study is to give the research community strong evidence about when and how AI can be safely and effectively used in systematic reviews to help summarize medical research.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Gartlehner G, Banda S, Callaghan M, et al. Cochrane evaluation of (semi-)automated review methods: protocol for an adaptive platform study within reviews. J Clin Epidemiol. 2026. PMID: 42320766. doi:10.1016/j.jclinepi.2026.112390

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。