文献速递流行病学与生物统计医学教育与科研方法医学人工智能与信息学

Cochrane Crowd众包识别随机试验准确性评估

An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials.
PubMed 文献 · J Clin Epidemiol · 2021-01-18

该研究评估Cochrane Crowd众包平台在识别随机试验中的准确性、产能与自主性,并与传统双重筛查的准金标准比较。结果显示各任务灵敏度97.7%–99.7%、特异度98.6%–99.1%,仅16.6%–19.7%记录需专家解决,联合机器学习 workflow 产能6年增5倍。

要点

  • 众包筛查识别RCT的灵敏度与特异度均高,可支持证据合成。
  • 多数记录无需专家复核,专家解决比例约16.6%–19.7%。
  • 众包联合机器学习 workflow 产能6年提升5倍。

核心结论

  • Cochrane Crowd的准确性、可扩展性足以跟上新原始研究的发表和注册速度。
  • 该平台已成为大量志愿者高效参与卫生证据生产的流行、准确方式。

研究设计与样本

  • 研究评估Cochrane Crowd在三个识别随机试验任务中的准确性、产能与自主性。
  • 准确性以众包筛查决定与同一记录传统双重筛查的准金标准决定比较,测量灵敏度和特异度。
  • 产能以众包处理的记录数相对于基线评估;自主性以需专家解决的记录比例测量。

主要发现

  • Cochrane Crowd社区现有来自163个国家的18,897名贡献者。
  • 众包累计处理1,021,227条记录,帮助识别178,437篇随机对照试验报告进入Cochrane CENTRAL。
  • RCT ID任务灵敏度99.1%、特异度99%;CT ID任务灵敏度99.7%、特异度98.6%;ICTRP ID任务灵敏度97.7%、特异度99.1%。
  • 众包与机器学习联合 workflow 的产能在6年内较基线提升5倍。
  • 各任务需专家解决的记录比例范围为16.6%至19.7%。

关键数值与推荐

  • RCT识别任务灵敏度99.1%、特异度99%。
  • ClinicalTrials.gov试验识别任务灵敏度99.7%、特异度98.6%。
  • ICTRP试验识别任务灵敏度97.7%、特异度99.1%。
  • 需专家解决的记录比例16.6%–19.7%。

临床怎么用

  • 可将众包筛查用于随机试验识别以加速证据合成中的记录筛选。
  • 众包联合机器学习 workflow 可提升筛选产能。
  • 多数众包决定可集体准确得出,仅少数记录需专家复核。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Noel-Storr A, Dooley G, Elliott J, et al. An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials. J Clin Epidemiol. 2021. PMID: 33476769. doi:10.1016/j.jclinepi.2021.01.006

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。