众包与机器学习用于系统评价文献筛选的准确性评估
本子研究评估Cochrane Screen4Me改良工作流(机器学习分类器结合Cochrane Crowd众包平台)筛选随机对照试验(RCT)的性能。112名参与者中81人完成培训并参与实际筛选,采用预设一致性算法,众包筛选达到100%敏感度和80.71%特异度,机器学习分类器未排除任何纳入研究。
要点
- 众包筛选联合一致性算法可达100%敏感度。
- 特异度为80.71%,可有效排除不相关记录。
- 机器学习分类器未误排任何纳入的RCT。
核心结论
- 使用众包筛选系统评价检索结果可成为准确方法,前提是一致性算法稳健。
研究设计与样本
- 子研究嵌套于一项大型随机研究,纳入未符合主研究随机分组条件的参与者。
- 记录经机器学习分类器分为「非RCT」与「可能RCT」,后者载入Cochrane Crowd平台由参与者判定「可能相关」或「不相关」。
- 采用预设一致性算法计算众包识别纳入研究的敏感度和排除未纳入研究的特异度。
主要发现
- RCT机器学习分类器未排除任何纳入研究。
- 112名参与者纳入子研究,其中81人完成培训模块并参与实际筛选任务。
- 应用Cochrane Crowd一致性算法,众包达到100%敏感度和80.71%特异度。
关键数值与推荐
- 众包筛选敏感度为100%,特异度为80.71%。
- 112名参与者中81人完成培训并进入实际筛选。
临床怎么用
- 在系统评价文献筛选流程中可考虑采用众包平台结合稳健一致性算法辅助识别RCT。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To assess the feasibility of a modified workflow that uses machine learning and crowdsourcing to identify studies for potential inclusion in a systematic review. STUDY DESIGN AND SETTING: This was a substudy to a larger randomized study; the main study sought to assess the performance of single screening search results versus dual screening. This substudy assessed the performance in identifying relevant randomized controlled trials (RCTs) for a published Cochrane review of a modified version of Cochrane's Screen4Me workflow which uses crowdsourcing and machine learning. We included participants who had signed up for the main study but who were not eligible to be randomized to the two main arms of that study. The records were put through the modified workflow where a machine learning classifier divided the data set into "Not RCTs" and "Possible RCTs." The records deemed "Possible RCTs" were then loaded into a task created on the Cochrane Crowd platform, and participants classified those records as either "Potentially relevant" or "Not relevant" to the review. Using a prespecified agreement algorithm, we calculated the performance of the crowd in correctly identifying the studies that were included in the review (sensitivity) and correctly rejecting those that were not included (specificity). RESULTS: The RCT machine learning classifier did not reject any of the included studies. In terms of the crowd, 112 participants were included in this substudy. Of these, 81 completed the training module and went on to screen records in the live task. Applying the Cochrane Crowd agreement algorithm, the crowd achieved 100% sensitivity and 80.71% specificity. CONCLUSIONS: Using a crowd to screen search results for systematic reviews can be an accurate method as long as the agreement algorithm in place is robust. TRIAL REGISTRATION: Open Science Framework: https://osf.io/3jyqt.
引用
- Noel-Storr A, Dooley G, Affengruber L, et al. Citation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service. J Clin Epidemiol. 2020. PMID: 33007457. doi:10.1016/j.jclinepi.2020.09.024
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。