单评审员评估网络Meta分析证据确定性可能不影响决策:案例研究
该案例研究基于COVID-19动态网络Meta分析,比较 novice 或专家单评审员与两人共识在GRADE证据确定性评级上的分歧。新手评审员总体分歧率高于专家(10.6% vs 2.3%),但重要分歧率相似(1.2% vs 1.0%),单评审员带来的重要分歧较少,可能不影响决策。
要点
- 新手单评审员总体分歧率高于专家,但重要分歧率与专家相近。
- 单评审员评估证据确定性的重要分歧率较低,对决策影响有限。
- 新手与专家发生重要分歧的关联无统计学显著性(OR 1.3,95%CI 0.9-1.8)。
核心结论
- 新手评审员评估网络Meta分析证据确定性时比专家评审员更常与共识分歧,但单评审员所致重要分歧较少,可能对决策者无实质影响。
研究设计与样本
- 方法学案例研究,使用COVID-19动态网络Meta分析的GRADE评估,由6名专家和5名新手评审员独立评估6652个比较的证据确定性,分歧通过共识解决。
主要发现
- 新手评审员总体证据确定性评级的分歧率高于专家(10.6% vs 2.3%),但重要分歧率相似(1.2% vs 1.0%)。
- 新手评审员在所有GRADE域(包括直接、间接和网络证据评估)的分歧率均高于专家。
- 新手评审员更可能发生重要分歧,但关联无统计学显著性(OR 1.3,95%CI 0.9-1.8)。
关键数值与推荐
- 重要分歧定义为证据确定性评级从高/中变为低/极低或反之。
- 新手评审员总体证据确定性评级分歧率10.6%,专家2.3%。
- 重要分歧率新手1.2%,专家1.0%。
- 新手与专家重要分歧的OR 1.3(95%CI 0.9-1.8)。
临床怎么用
- 在资源有限时,单评审员评估网络Meta分析证据确定性可能足以支持决策,重要分歧较少。
- 新手评审员可参与评估,但需注意其总体分歧率较高,重要分歧与专家无显著差异。
局限性
- 研究仅基于COVID-19动态网络Meta分析,结论外推需谨慎。
- 新手和专家评审员数量有限(6名专家和5名新手)。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Systematic reviews with network meta-analyses (NMA) are the most comprehensive form of evidence synthesis for informing evidence-based practice recommendations. Rating the certainty of evidence (CoE) is a crucial step for drawing appropriate NMA's conclusions, but it can be time-consuming. OBJECTIVES: To evaluate the disagreements of assessments by a single reviewer (novice or expert) v consensus among two reviewers on CoE ratings of NMA used to support practice recommendations. METHODS: We conducted a methodological case study using the grading of recommendations, assessment, development, and evaluations (GRADE) assessments for each comparison and outcome of interest from the COVID-19 living NMA, which informed the World Health Organization recommendations. Expert and novice reviewers assessed CoE independently and resolved disagreements by consensus. We defined an "important disagreement" in CoE ratings as any disagreement that would lead to an increase from high/moderate to low/very low or vice versa. We calculated the proportion of outcomes with important disagreements comparing the CoE rating from a single vs two reviewers. We performed a generalized linear mixed model analysis to compare the relative odds of disagreement between expert and novice reviewers. RESULTS: Six experts and five novice reviewers rated 6652 comparisons. Compared with consensus ratings, novice reviewers exhibited higher disagreement rates across all GRADE domains, including assessments of direct, indirect, and network evidence. The disagreement rates for the overall CoE rating assessed by novice reviewers were higher than those of experts (10.6 vs 2.3%), but the important disagreement rate was similar (1.2% vs 1.0%). Novice reviewers were more likely to make important disagreements, but the association was not statistically significant (odds ratio, 1.3; 95% confidence interval 0.9-1.8). CONCLUSION: Novice reviewers disagreed with the consensus more often than expert reviewers when assessing the CoE from NMA. However, having a single reviewer was associated with relatively few disagreements likely to have meaningful implications for decision-makers.
引用
- Zaror C, Chávez-Concha R, Izcovich A, et al. Single reviewer assessments of certainty of evidence in network meta-analyses probably do not mislead decision-makers: a case study. J Clin Epidemiol. 2026. PMID: 42551807. doi:10.1016/j.jclinepi.2026.112441
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。