Cochrane随机对照试验分类器的开发与评估
本研究开发并校准了一种机器学习分类器(Cochrane RCT分类器),旨在减少Cochrane系统评价中研究识别的筛选工作量。该分类器使用Cochrane Crowd人工标注的Embase题录摘要数据训练,并以99%召回率为目标用Clinical Hedges团队标注数据进行校准;最终在Cochrane综述纳入的RCT记录中评估,分类器正确检索了99.5%的RCT,仅遗漏0.5%,且较老记录更易被遗漏。
要点
- Cochrane RCT分类器可显著减少系统评价中研究识别的人工筛选工作量。
- 校准后分类器在评估数据集中召回率为99.5%,遗漏RCT风险低且可接受。
- 较老记录的RCT更可能被分类器遗漏。
核心结论
- Cochrane RCT分类器可减少Cochrane系统评价中人工研究识别的工作量,且遗漏合格RCT的风险非常低、可接受。
研究设计与样本
- 开发并校准用于检索随机对照试验的机器学习分类器,训练集为Cochrane Crowd人工标注的Embase题录摘要记录,校准集为Clinical Hedges团队人工标注的记录。
- 使用Cochrane综述纳入的、摘要长度足够的RCT记录评估校准后分类器的召回率。
主要发现
- Cochrane RCT分类器训练使用280,620条记录,其中20,454条报告RCT。
- 校准使用49,025条记录(其中1,587条报告RCT),bootstrap验证显示召回率0.99(95%CI 0.98-0.99),精确率0.08(95%CI 0.06-0.12)。
- 最终校准分类器正确检索44,007条Cochrane综述纳入RCT中的43,783条(99.5%),遗漏224条(0.5%)。
- 较老记录比近期发表的记录更可能被遗漏。
临床怎么用
- 该分类器已作为Evidence Pipeline的一部分部署在Cochrane内,用于提升系统评价制作中研究识别流程的效率。
局限性
- 评估仅纳入摘要长度足以进行机器分类的RCT记录。
- 较老记录更易被分类器遗漏。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. METHODS: A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the "Cochrane RCT Classifier"), with the algorithm trained using a data set of title-abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. RESULTS: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98-0.99) and precision of 0.08 (95% confidence interval 0.06-0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. CONCLUSIONS: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.
引用
- Thomas J, McDonald S, Noel-Storr A, et al. Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews. J Clin Epidemiol. 2020. PMID: 33171275. doi:10.1016/j.jclinepi.2020.11.003
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。