RobotReviewer机器学习工具辅助系统评价偏倚风险评估的前瞻性横断面评价
本研究前瞻性横断面评价RobotReviewer对1,180项试验的偏倚风险判断与人工评审者之间的一致性,采用Cohen kappa系数计算信度并计算敏感度与特异度。结果显示随机序列生成、分配隐藏与参与者及人员盲法的信度中等,总体偏倚风险为一般,结局评估者盲法、不完整结局数据与选择性报告的信度仅为轻微。结论认为偏倚风险评估具有主观性,RobotReviewer在多数领域与人工评审者的一致性相似,在分配隐藏、参与者及人员盲法与总体偏倚风险方面更好。
要点
- RobotReviewer与人工评审者在随机序列生成、分配隐藏、参与者及人员盲法领域信度中等。
- 结局评估者盲法、不完整结局数据、选择性报告领域信度仅为轻微。
- 部分领域信度按主题不同存在差异。
核心结论
- 偏倚风险评估具有主观性;与作者组间信度相比,RobotReviewer与人工评审者的信度在多数领域相似,在分配隐藏、参与者及人员盲法和总体偏倚风险方面更好。
研究设计与样本
- 前瞻性横断面评价,使用RobotReviewer评估1,180项试验的偏倚风险,以Cohen kappa系数计算与人工评审者的信度,并计算敏感度与特异度,采用meta分析中的卡方检验考察不同偏倚风险领域、主题与结局类型的信度差异。
主要发现
- 随机序列生成信度0.48(95%CI 0.43–0.53)、分配隐藏0.45(0.40–0.51)、参与者及人员盲法0.42(0.36–0.47)为中等;总体偏倚风险0.34(0.25–0.44)为一般;结局评估者盲法0.10(0.06–0.14)、不完整结局数据0.14(0.08–0.19)、选择性报告0.02(-0.02–0.05)为轻微。
- 参与者及人员盲法(P < 0.001)、结局评估者盲法(P = 0.005)、选择性报告(P < 0.001)和总体偏倚风险(P < 0.001)的信度按主题存在差异。
- 敏感度(95%CI)范围为0.20(0.18–0.23)至0.76(0.72–0.80),特异度(95%CI)范围为0.61(0.56–0.65)至0.95(0.93–0.96)。
局限性
- 原文未述局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To evaluate the reliability of RobotReviewer's risk of bias judgments. STUDY DESIGN AND SETTING: In this prospective cross-sectional evaluation, we used RobotReviewer to assess risk of bias among 1,180 trials. We computed reliability with human reviewers using Cohen's kappa coefficient and calculated sensitivity and specificity. We investigated differences in reliability by risk of bias domain, topic, and outcome type using the chi-square test in meta-analysis. RESULTS: Reliability (95% CI) was moderate for random sequence generation (0.48 [0.43, 0.53]), allocation concealment (0.45 [0.40, 0.51]), and blinding of participants and personnel (0.42 [0.36, 0.47]); fair for overall risk of bias (0.34 [0.25, 0.44]); and slight for blinding of outcome assessors (0.10 [0.06, 0.14]), incomplete outcome data (0.14 [0.08, 0.19]), and selective reporting (0.02 [-0.02, 0.05]). Reliability for blinding of participants and personnel (P < 0.001), blinding of outcome assessors (P = 0.005), selective reporting (P < 0.001), and overall risk of bias (P < 0.001) differed by topic. Sensitivity and specificity (95% CI) ranged from 0.20 (0.18, 0.23) to 0.76 (0.72, 0.80) and from 0.61 (0.56, 0.65) to 0.95 (0.93, 0.96), respectively. CONCLUSION: Risk of bias appraisal is subjective. Compared with reliability between author groups, RobotReviewer's reliability with human reviewers was similar for most domains and better for allocation concealment, blinding of participants and personnel, and overall risk of bias.
引用
- Gates A, Vandermeer B, Hartling L, et al. Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool. J Clin Epidemiol. 2017. PMID: 29289761. doi:10.1016/j.jclinepi.2017.12.015
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。