文献速递流行病学与生物统计

使用预测模型偏倚风险评估工具(PROBAST)评估医疗预测模型质量:对其使用和实际应用的评价

Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application.
PubMed 文献 · J Clin Epidemiol · 2025-02-25

该案例研究基于对诊断和预后COVID-19预测模型系统综述的PROBAST评估,分析了27对评估者共2167份评估(覆盖760个预测模型)的评估者间信度,发现总体偏倚风险判断的PABAK较高(开发0.82,验证0.78),共识讨论主要改变条目级而非总体评级。

要点

  • 总体偏倚风险判断的评估者间信度高于条目和领域水平。
  • 条目3.5结局盲法和4.1样本量常影响领域偏倚风险判断。
  • 共识讨论主要导致条目级评级变化,未改变总体偏倚风险评级。

核心结论

  • 在本案例研究中,PROBAST评估在总体偏倚风险水平上获得高评估者间信度,条目和领域水平存在一些变异。
  • 为减少变异性,PROBAST评估者应标准化条目和领域水平的判断,并在同一研究的评估者之间举行结构良好的共识会议。

研究设计与样本

  • 研究设计为案例研究,使用对诊断和预后COVID-19预测模型系统综述的PROBAST评估。
  • 评估者包括预测模型研究或其综述的国际专家。
  • 分析了27对评估者的2167份PROBAST评估,覆盖760个预测模型:384个开发、242个验证和134个混合评估(包括两者)。
  • 在共识会议前使用PABAK评估评估者间信度,检查每个领域偏倚风险判断的偏倚评级,并通过识别讨论后的评级变化评估共识会议的影响。

主要发现

  • 一些PROBAST条目频繁贡献于领域水平偏倚风险判断,例如3.5结局盲法和4.1样本量。
  • 共识讨论主要导致条目水平变化,从未导致总体偏倚风险评级变化。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND AND OBJECTIVES: Since 2019, the Prediction model Risk Of Bias ASsessment Tool (PROBAST; www.probast.org) has supported methodological quality assessments of prediction model studies. Most prediction model studies are rated with a "High" risk of bias (ROB) and researchers report low interrater reliability (IRR) using PROBAST. We aimed to (1) assess the IRR of PROBAST ratings between assessors of the same study and understand reasons for discrepancies, (2) determine which items contribute most to domain-level ROB ratings, and (3) explore the impact of consensus meetings. STUDY DESIGN AND SETTING: We used PROBAST assessments from a systematic review of diagnostic and prognostic COVID-19 prediction models as a case study. Assessors included international experts in prediction model studies or their reviews. We assessed IRR using prevalence-adjusted bias-adjusted kappa (PABAK) before consensus meetings, examined bias ratings per domain-level ROB judgments, and evaluated the impact of consensus meetings by identifying rating changes after discussion. RESULTS: We analyzed 2167 PROBAST assessments from 27 assessor pairs covering 760 prediction models: 384 developments, 242 validations, and 134 mixed assessments (including both). The IRR using PABAK was higher for overall ROB judgments (development: 0.82 [0.76; 0.89]; validation: 0.78 [0.68; 0.88]) compared to domain- and item-level judgments. Some PROBAST items frequently contributed to domain-level ROB judgments, eg, 3.5 Outcome blinding and 4.1 Sample size. Consensus discussions mainly led to item-level and never to overall ROB rating changes. CONCLUSION: Within this case study, PROBAST assessments received high IRR at the overall ROB level, with some variation at item- and domain-level. To reduce variability, PROBAST assessors should standardize item- and domain-level judgments and hold well-structured consensus meetings between assessors of the same study. PLAIN LANGUAGE SUMMARY: The Prediction model Risk Of Bias ASsessment Tool (PROBAST; www.probast.org) provides a set of items to assess the quality of medical studies on so-called prediction tools that calculate an individual's probability of having or developing a certain disease or health outcome. Previous research found low interrater reliability (IRR; ie, how consistently two assessors rate aspects of the same study) when using PROBAST. To understand why this is the case, we conducted a large study involving more than 30 experts from around the world, all of whom applied PROBAST to the same set of prediction tool studies. Based on more than 2150 PROBAST assessments, we identified which PROBAST items led to the most disagreements between raters, explored reasons for these disagreements, and examined whether the use of so-called consensus meetings (ie, different assessors of the same study discuss their ratings and decide on a finalized rating) impacted PROBAST ratings. Our study found that the IRR between different assessors of the same study was higher than previously reported. One explanation for the better agreement compared to previous research may be the preplanning on how to assess certain PROBAST aspects before starting the assessments, as well as holding well-structured consensus meetings. These improvements lead to a more effective use of PROBAST in evaluating the trustworthiness and quality of prediction tools in the health-care domain.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Kaul T, Damen JAA, Wynants L, et al. Assessing the quality of prediction models in health care using the Prediction model Risk Of Bias ASsessment Tool (PROBAST): an evaluation of its use and practical application. J Clin Epidemiol. 2025. PMID: 40010583. doi:10.1016/j.jclinepi.2025.111732

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。