众包批判性评价研究证据(CrowdCARE)被证实是评估临床研究质量的有效方法
本研究开发了免费在线工具CrowdCARE,用于众包研究批判性评价,并检验其评估系统综述方法学质量的有效性。前瞻性横断面研究中,71篇异质性系统综述由五名受训新手和两名专家使用AMSTAR工具独立评价,专家共识作为金标准。结果显示专家共识评分与新手平均评分强相关(r²=0.89,P<0.0001)且一致性很好(κ=0.67,95%CI:0.61-0.73),表明众包可用于评估系统综述质量。
要点
- 众包可用于评估系统综述的方法学质量。
- 受训新手平均评分与专家共识高度相关且一致性良好。
- 该概念验证数据支持众包替代当前金标准的潜力。
核心结论
- 众包可以用于评估系统综述的质量,受训新手平均能达到与专家相比的高准确度。
研究设计与样本
- 前瞻性横断面研究,样本为71篇异质性系统综述,由五名受训新手和两名专家使用AMSTAR工具在CrowdCARE中独立评价,专家共识作为金标准。
主要发现
- 专家共识评分与个体专家评分在82%的研究中相差在±1以内(11分制)。
- 专家共识评分与新手平均评分在79%的研究中相差在±1以内。
- 专家共识评分与新手平均评分强相关(r²=0.89,P<0.0001),一致性很好(κ=0.67,95%CI:0.61-0.73)。
关键数值与推荐
- 专家共识评分与个体专家评分在82%的研究中相差在±1以内(11分制)。
- 专家共识评分与新手平均评分在79%的研究中相差在±1以内。
- 专家共识评分与新手平均评分强相关(r²=0.89,P<0.0001),一致性很好(κ=0.67,95%CI:0.61-0.73)。
临床怎么用
- 可训练新手使用CrowdCARE众包评价系统综述质量,以分担评价负荷。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: We developed a free, online tool (CrowdCARE: crowdcare.unimelb.edu.au) to crowdsource research critical appraisal. The aim was to examine the validity of this approach for assessing the methodological quality of systematic reviews. STUDY DESIGN AND SETTING: In this prospective, cross-sectional study, a sample of systematic reviews (N = 71), of heterogeneous quality, was critically appraised using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool, in CrowdCARE, by five trained novice and two expert raters. After performing independent appraisals, experts resolved any disagreements by consensus (to produce an "expert consensus" rating, as the gold-standard approach). RESULTS: The expert consensus rating was within ±1 (on an 11-point scale) of the individual expert ratings for 82% of studies and was within ±1 of the mean novice rating for 79% of studies. There was a strong correlation (r2 = 0.89, P < 0.0001) and very good concordance (κ = 0.67, 95% CI: 0.61-0.73) between the expert consensus rating and mean novice rating. CONCLUSION: Crowdsourcing can be used to assess the quality of systematic reviews. Novices can be trained to appraise systematic reviews and, on average, achieve a high degree of accuracy relative to experts. These proof-of-concept data demonstrate the merit of crowdsourcing, compared with current gold standards of appraisal, and the potential capacity for this approach to transform evidence-based practice worldwide by sharing the appraisal load.
引用
- Pianta MJ, Makrai E, Verspoor KM, et al. Crowdsourcing critical appraisal of research evidence (CrowdCARE) was found to be a valid approach to assessing clinical research quality. J Clin Epidemiol. 2018. PMID: 30075189. doi:10.1016/j.jclinepi.2018.07.015
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。