单审稿人与双审稿人筛选标题摘要准确性的比较:快速综述方法学权衡的元研究第一部分
该元研究在3篇关于慢性原发性下背痛干预措施有效性的综述中,比较了1名审稿人与2名审稿人独立筛选标题和摘要后进入全文筛选的准确性。单审稿人敏感性为48.8%至66.3%,特异性为88.0%至93.3%,5.0%至6.3%的记录被错误分类为假阴性。结论提示单审稿人筛选可能排除合格记录,不建议仅用Kappa作为筛选质量指标。
要点
- 单审稿人筛选标题摘要的敏感性为48.8%至66.3%,可能遗漏合格记录。
- 单审稿人筛选特异性为88.0%至93.3%,排除无关记录的能力较好。
- 5.0%至6.3%的记录被单审稿人错误分类为假阴性,主要涉及干预和对照判断。
核心结论
- 单审稿人筛选标题和摘要可能导致合格记录在快速综述中被排除。
- 不建议仅用Kappa作为筛选质量的指标,因其受分类不平衡影响。
- 建议纳入准确性指标来描述审稿人筛选分类差异的潜在可能。
研究设计与样本
- 在3篇关于慢性原发性下背痛干预措施有效性的综述中比较单审稿人与双审稿人共识筛选。
- 单审稿人使用标准化合格标准筛选标题和摘要,结果与双审稿人共识比较。
- 计算敏感性、特异性、阳性预测值和阴性预测值及其95%置信区间。
主要发现
- 单审稿人敏感性范围为48.8%至66.3%,特异性范围为88.0%至93.3%。
- 阳性预测值范围为40.6%至51.8%,阴性预测值范围为93.6%至95%。
- 审稿人间信度范围为0.39至0.50。
- 5.0%至6.3%的记录被单审稿人错误分类为假阴性。
- 错误分类原因主要与相关干预和对照的评估有关,例如干预是否能被隔离。
关键数值与推荐
- 仅筛选英文记录与所有语言记录相比,敏感性和阳性预测值提高,特异性和阴性预测值无变化。
临床怎么用
- 在快速综述中应谨慎使用单审稿人筛选标题和摘要,因其可能遗漏合格记录。
- 建议使用额外的质量保证措施来限制单审稿人筛选带来的偏倚。
- 评估筛选质量时应结合准确性指标,而非仅依赖Kappa值。
局限性
- 研究仅基于3篇关于慢性原发性下背痛干预措施有效性的综述。
- 因使用人工智能将中文研究翻译为英文,该方法的有用性需进一步研究。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To determine the accuracy for progressing records to full-text screening using one vs two reviewers to screen title and abstracts in 3 reviews of the effectiveness of interventions for chronic primary low back pain. Secondary objectives include computing inter-rater reliability, describing misclassified records and reviewer performance across reviews, and conducting sensitivity analysis limited to English records and falsely excluded records. STUDY DESIGN AND SETTING: One reviewer screened title and abstracts using standardized eligibility criteria and results were compared to consensus screening from two reviewers. We computed sensitivity, specificity, positive (PPV) and negative predictive values (NPV) with 95% confidence intervals using the two reviewers as the comparison. We calculated the inter-rater reliability, proportion of misclassified citations, and the reasons for misclassification. We conducted sensitivity analyses by restricting the analysis to English records. RESULTS: The sensitivity of one reviewer ranged from 48.8% to 66.3% and the specificity ranged from 88.0% to 93.3%. The PPV ranged from 40.6% to 51.8% and NPV 93.6% to 95%. The inter-rater reliability ranged from 0.39 to 0.50. Between 5.0% and 6.3% of records were misclassified as false negative by a single reviewer. Reasons for misclassification were primarily related to the assessment of relevant interventions and comparators, such as whether the intervention could be isolated. Our sensitivity analysis showed that screening English records only compared to all languages improved sensitivity and PPV, with no change in specificity and NPV. CONCLUSION: Using a single reviewer to screen titles and abstracts may lead to the exclusion of eligible records during title and abstract screening in rapid reviews of the literature. We caution against using Kappa alone as an indicator of the quality of screening, as it is influenced by classification imbalances and suggest including accuracy measures to describe the potential for differences between reviewer screening classifications. PLAIN LANGUAGE SUMMARY: This study investigated whether one reviewer can accurately screen research articles for inclusion in a systematic review, compared to the usual approach of having 2 people do the screening. This was tested in three reviews of common treatments for chronic primary low back pain. The single reviewer who screened titles and abstracts was likely to miss relevant articles that were identified as relevant by 2 reviewers. However, the single reviewer was good at correctly excluding irrelevant articles. Between 5% and 6% of eligible articles were incorrectly excluded by the single reviewer. Most mistakes happened when the single reviewer was uncertain about a treatment's eligibility. Limiting screening to English language articles slightly improved the accuracy of the screening but it did not eliminate the risk of missing relevant research. Since artificial intelligence was used to translate Chinese studies to English, further research on the usefulness for this approach is warranted. In summary, restricting screening of articles to one reviewer may save time, but it increases the probability that important evidence will be overlooked. Researchers should be cautious about relying on a single reviewer and should use additional quality assurance to limit bias.
引用
- Corso M, Ead L, Brunton G, et al. Evaluating the accuracy of screening titles and abstracts for progressing to full-text screening by one vs two independent reviewers: part 1 of a meta-research study of rapid review methodological trade-offs. J Clin Epidemiol. 2026. PMID: 41865804. doi:10.1016/j.jclinepi.2026.112248
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。