众包与自动化助力在动态系统评价中识别和分类随机对照试验
该研究评估在类风湿关节炎动态系统评价中,联合机器学习和Cochrane Crowd众包初筛、实习审稿人用PICO平台分类、专家仲裁的流程。从42452条记录中排除非随机对照试验后,1946条记录对应220项随机对照试验,涵盖既往评价中148/149(99.3%)已知合格试验。该流程大幅减少专家审稿负担且敏感性高。
要点
- 机器学习联合Cochrane Crowd从42452条记录中排除28777条(68%)。
- 实习审稿人排除4529条(11%),专家排除7200条(17%)。
- 最终1946条记录对应220项随机对照试验,涵盖已知合格试验的148/149(99.3%)。
核心结论
- 联合机器学习、众包和实习审稿人参与的筛查流程大幅减少专家审稿负担且敏感性高。
研究设计与样本
- 在类风湿关节炎动态系统评价中评估自动化和众包识别与分类随机对照试验的方法。
- 数据库检索随机对照试验记录,先由机器学习和Cochrane Crowd排除非随机对照试验,再由实习审稿人使用PICO注释平台评估合格性并分类至相应评价,分歧由专家通过定制在线工具解决。
主要发现
- 从42452条记录中,机器学习和Cochrane Crowd排除28777条(68%),实习审稿人排除4529条(11%),专家排除7200条(17%)。
- 最终1946条记录对应220项随机对照试验,涵盖既往评价中已知合格试验的148/149(99.3%)。
- 6420条记录虽被排除于本动态系统评价,但被分类为类风湿关节炎的其他随机对照试验,供未来评价使用。
- 实习审稿人的假阴性率在随机对照试验域最高,为12%,其中仅1.1%涉及主要记录。
- 两名审稿人间的kappa评分为0.40-0.69,一致性从中等至相当。
局限性
- 实习审稿人的假阴性率在随机对照试验域最高,为12%,其中仅1.1%涉及主要记录。
- 两名审稿人间的kappa评分为0.40-0.69,一致性从中等至相当。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To evaluate an approach using automation and crowdsourcing to identify and classify randomized controlled trials (RCTs) for rheumatoid arthritis (RA) in a living systematic review (LSR). METHODS: Records from a database search for RCTs in RA were screened first by machine learning and Cochrane Crowd to exclude non-RCTs, then by trainee reviewers using a Population, Intervention, Comparison, and Outcome (PICO) annotator platform to assess eligibility and classify the trial to the appropriate review. Disagreements were resolved by experts using a custom online tool. We evaluated the efficiency gains, sensitivity, accuracy, and interrater agreement (kappa scores) between reviewers. RESULTS: From 42,452 records, machine learning and Cochrane Crowd excluded 28,777 (68%), trainee reviewers excluded 4,529 (11%), and experts excluded 7,200 (17%). The 1,946 records eligible for our LSR represented 220 RCTs and included 148/149 (99.3%) of known eligible trials from prior reviews. Although excluded from our LSRs, 6,420 records were classified as other RCTs in RA to inform future reviews. False negative rates among trainees were highest for the RCT domain (12%), although only 1.1% of these were for the primary record. Kappa scores for two reviewers ranged from moderate to substantial agreement (0.40-0.69). CONCLUSION: A screening approach combining machine learning, crowdsourcing, and trainee participation substantially reduced the screening burden for expert reviewers and was highly sensitive.
引用
- Kamso MM, Pardo JP, Whittle SL, et al. Crowd-sourcing and automation facilitated the identification and classification of randomized controlled trials in a living review. J Clin Epidemiol. 2023. PMID: 37865299. doi:10.1016/j.jclinepi.2023.10.007
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。