文献速递流行病学与生物统计医学教育与科研方法

单 vs 双独立评审者全文与序贯筛选准确性:快速综述方法学权衡的元研究(第2部分)

Evaluating the accuracy of full text and serial screening by one vs two independent reviewers: part 2 of a meta-research study of rapid review methodological trade-offs.
PubMed 文献 · J Clin Epidemiol · 2026-09-28

该元研究基于针刺、教育和TENS三项慢性腰背痛干预系统综述的全文与序贯筛选数据,比较单评审者与双评审者共识决策的准确性。单评审者全文筛选敏感度62%–88%、特异度84%–96%,序贯筛选敏感度62%–81%、特异度98%–100%、NPV 99%;Kappa 0.61–0.73。序贯筛选阳性PTP为42.7%–63.5%,错误分类主要源于人群、干预或对照解读差异。

要点

  • 单评审者全文筛选敏感度62%–88%,可能漏选应纳入研究。
  • 单评审者序贯筛选NPV达99%,但阳性PTP仅42.7%–63.5%。
  • 错误分类多因人群、干预或对照的解读差异,需清晰定义筛选标准。

核心结论

  • 单评审者序贯筛选准确性可能降低,导致漏选和误纳研究,且各综述间存在差异。
  • 单评审者能否正确选择相关研究仍存在不确定性,但对快速综述结果的影响尚不清楚。
  • 总体一致性为中等至高度,报告敏感度和特异度有助于了解评审者间的分类差异。

研究设计与样本

  • 一项元研究,基于三项评估慢性腰背痛干预(针刺、教育、TENS)的系统综述。
  • 由一名评审者筛选全文,并与同一评审者筛选的题录/摘要结合评估序贯筛选准确性。
  • 以两名独立评审者的共识决策为参照,计算敏感度、特异度、PPV、NPV及Cohen kappa。
  • 针刺、教育和TENS综述分别筛选了120、92和91篇研究。
  • 对非英语研究进行AI辅助翻译的敏感性分析,比较英语与所有语言分析的估计差异。

主要发现

  • 单评审者全文筛选敏感度62%–88%、特异度84%–96%、PPV 70%–77%、NPV 89%–97%,Kappa 0.61–0.73。
  • 单评审者序贯筛选敏感度62%–81%、特异度98%–100%、PPV 68%–71%、NPV 99%,Kappa 0.67–0.73。
  • 序贯筛选阳性PTP为42.7%–63.5%。
  • 错误分类主要源于对人群、干预或对照的解读差异。
  • 英语与所有语言分析的估计无差异或差异很小,置信区间重叠。

关键数值与推荐

  • 单评审者全文筛选敏感度62%–88%、特异度84%–96%、PPV 70%–77%、NPV 89%–97%。
  • 单评审者序贯筛选敏感度62%–81%、特异度98%–100%、PPV 68%–71%、NPV 99%。
  • 全文筛选Kappa 0.61–0.73,序贯筛选Kappa 0.67–0.73。
  • 序贯筛选阳性PTP范围42.7%–63.5%。

临床怎么用

  • 使用单评审者进行快速综述筛选可节省时间,但需认识到可能漏选应纳入研究。
  • 筛选时应清晰定义人群、干预和对照,以减少解读差异导致的错误分类。
  • 报告敏感度和特异度可提供评审者间分类差异的额外信息。
  • AI辅助翻译非英语研究未对准确性估计产生明显影响,可作为筛选辅助工具。

局限性

  • 单评审者筛选准确性降低对快速综述结果的影响尚不清楚。
  • 研究仅基于三项慢性腰背痛干预的系统综述,结论外推需谨慎。
  • 未评估其他疾病领域或不同筛选流程下的准确性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: We aimed to determine the accuracy of one vs two reviewers for full-text screening and serial screening (title/abstract and full text). We also determined the inter-rater reliability, post-test probability (PTP), false positive and negative rates, and sources of misclassification. STUDY DESIGN AND SETTING: One reviewer screened full-text articles from three systematic reviews evaluating interventions (acupuncture, education, and transcutaneous electrical nerve stimulation [TENS]) for chronic low back pain. We combined these results with titles and abstracts screened by the same reviewer to assess serial screening accuracy (screening title and abstracts, then full-text studies). Outcomes were compared to consensus decisions from two independent reviewers. We computed sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) with 95% confidence intervals (CIs). Inter-rater reliability (Cohen's kappa) and positive PTP for serial screening were calculated. Rates and reasons for misclassification were examined. A sensitivity analysis of English studies explored the impact of AI-assisted translation of non-English studies. RESULTS: A total of 120, 92, and 91 studies were screened for the acupuncture, education, and TENS reviews, respectively. Compared with two reviewers, the sensitivity of single reviewer full-text screening ranged from 62% to 88%, specificity from 84% to 96%, PPV from 70% to 77%, and NPV from 89% to 97%. Kappa ranged from 0.61 to 0.73. Single-reviewer sensitivity for serial screening ranged from 62% to 81%, specificity from 98% to 100%, PPV from 68% to 71%, and NPV was 99%. Kappa ranged from 0.67 to 0.73. Positive PTP ranged from 42.7% to 63.5%. Misclassifications were mostly due to interpretation of the population, interventions, or comparators. There were no or small differences in the estimates with overlap of CIs between English and all language analyses. CONCLUSION: The accuracy of one vs two reviewers may be reduced in serial screening, resulting in missed and falsely included studies, ranging across reviews. Therefore, there is uncertainty around whether a single reviewer can correctly select relevant studies. However, the impact on rapid review results is unknown. Although overall agreement is substantial, reporting sensitivity and specificity can provide insight into classifications between reviewers. Carefully defining the population, intervention, and comparators in screening is required and highlights the need for improved reporting in trials. PLAIN LANGUAGE SUMMARY: Systematic reviews use two steps to choose which studies should be included in the synthesis. Each step is typically done by two people, but using one reviewer can save time and money. This study investigated how one reviewer compares to two reviewers when deciding which studies to include in a review. The researchers used data from three systematic reviews of treatments for chronic low back pain (acupuncture, education, and TENS). One reviewer screened the studies to determine which should be included, and the results were compared to the original systematic reviews which involved two reviewers. Researchers measured the accuracy of one reviewer compared to two, what caused mistakes and whether using AI to translate studies that were not in English affected accuracy. The single reviewer was generally correct when selecting studies. However, the single reviewer also missed studies that should have been included. Most errors happened because of different interpretations of whether participants had chronic low back pain and the identification of treatments. There were no differences in accuracy when non-English studies were translated with AI. Overall, using one reviewer to select studies for a review saves time. However, they can miss studies that should be included in the review. Clear understanding of study parameters is essential to reduce errors, and better reporting is needed to help select studies for reviews. AI is a useful tool for translating studies for study selection in a review.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Corso M, Ead L, Brunton G, et al. Evaluating the accuracy of full text and serial screening by one vs two independent reviewers: part 2 of a meta-research study of rapid review methodological trade-offs. J Clin Epidemiol. 2026. PMID: 42805513. doi:10.1016/j.jclinepi.2026.112542

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。