文献速递医学人工智能与信息学流行病学与生物统计

基于机器学习的急性胰腺炎预后模型系统评价:旨在改进方法与报告质量

A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality.
PubMed 文献 · PLoS Med · 2025-02-24

该系统评价检索MEDLINE、EMBASE及预印本服务器,纳入2021年1月至2023年12月发表的30项开发或验证急性胰腺炎机器学习预后模型的研究,共产生39个新模型,但所有模型至少一个领域存在高偏倚风险,报告仅覆盖TRIPOD+AI的15/27个条目,方法与报告质量存在显著缺陷。

要点

  • 30项研究共开发39个新模型,无一验证既有模型,预测结局以急性胰腺炎严重程度(23/39)或死亡(6/39)为主。
  • 所有39个模型至少一个领域偏倚风险高,其中分析领域为37/39;27/39未采取措施减少过于乐观的模型性能。
  • 报告仅覆盖TRIPOD+AI标准27个条目中的15个,仅7/30说明样本量依据、13/30评估数据质量。

核心结论

  • 近期发表的急性胰腺炎机器学习预后模型在方法与报告上存在显著缺陷,削弱了其有效性、可重复性与临床实施,尽管其预测准确性看似更优。

研究设计与样本

  • 系统评价,检索MEDLINE与EMBASE(2021年1月至2023年12月)及medRxiv、bioRxiv、arXiv预印本(2021年1月至2023年12月注册),纳入开发或验证急性胰腺炎机器学习预后模型的所有回顾性或前瞻性研究,共30项符合标准。
  • 纳入研究来自中国(22项)、美国(4项)和其他地区(4项);30项研究均开发新模型,共产生39个新机器学习模型。

主要发现

  • 所有模型与结局的平均曲线下面积为0.91(标准差0.08)。
  • 所有39个模型至少一个领域偏倚风险高,分析领域偏倚风险高者为37/39;27/39未采取措施减少过于乐观的模型性能。
  • 因研究设计及结局定义与判定存在异质性,未进行Meta分析。
  • 研究仅报告TRIPOD+AI标准27个条目中的15个,仅7/30说明样本量依据、13/30评估数据质量。
  • 报告缺陷包括未报告人机交互(28/30)、实践中低质量或不完整数据的处理(27/30)、分析代码共享(25/30)、研究方案(25/30)和源数据报告(19/30)。

关键数值与推荐

  • 偏倚风险评估采用预测模型偏倚风险评估工具(PROBAST)。
  • 报告质量评估采用TRIPOD+AI声明,该声明定义机器学习预后模型论文应报告的27个条目标准。
  • 预测结局以急性胰腺炎严重程度(23/39)或死亡(6/39)最常见。

局限性

  • 因研究设计及结局定义与判定存在异质性,未进行Meta分析。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: An accurate prognostic tool is essential to aid clinical decision-making (e.g., patient triage) and to advance personalized medicine. However, such a prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However, methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical for model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models. METHODS/FINDINGS: Using a validated search strategy, we identified ML AP studies from the databases MEDLINE and EMBASE published between January 2021 and December 2023. We also searched pre-print servers medRxiv, bioRxiv, and arXiv for pre-prints registered between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool. Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis-Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models. The search strategy identified 6,480 publications of which 30 met the eligibility criteria. Studies originated from China (22), the United States (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area under the curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed. Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human-AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30), and reporting source data (19/30). CONCLUSIONS: There are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility, and implementation of these prognostic models despite their promise of superior predictive accuracy. REGISTRATION: Research Registry (reviewregistry1727).

来源:PLoS Med,查看原文。版权归原作者所有。

引用

  1. Critelli B, Hassan A, Lahooti I, et al. A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. PLoS Med. 2025. PMID: 39992936. doi:10.1371/journal.pmed.1004432

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。