机器学习与逻辑回归妊娠期糖尿病预后模型的评价
本研究利用2021至2022年Monash Health网络12,722例单胎妊娠数据,对现有妊娠期糖尿病预测模型进行时间验证、更新并比较机器学习与回归模型的性能。结果显示各模型区分度相似,AUC为0.72至0.73,所有模型均出现过估计,重新校准后改善;决策曲线分析显示所有模型净获益优于全部治疗与不治疗策略,但机器学习原模型验证时性能显著下降。
要点
- 时间验证数据集纳入12,722例单胎妊娠,妊娠期糖尿病患病率为28.6%。
- 各模型区分度相似,AUC为0.72至0.73,所有模型均出现过估计。
- 重新校准可改善模型性能,所有模型净获益优于全部治疗与不治疗策略。
核心结论
- 尽管人群背景特征发生显著变化,所有模型仍保持稳健,尤其在重新校准后。
- 原始机器学习模型在验证期间性能显著下降。
- 动态模型更适合适应孕妇基线特征的时间变化及由此产生的校准漂移,因其可纳入新数据而无需手动评估。
研究设计与样本
- 研究采用2021至2022年Monash Health网络12,722例单胎妊娠的新数据作为时间验证数据集。
- 评估了Monash妊娠期糖尿病逻辑回归模型第2版(含6个分类变量)、Monash妊娠期糖尿病机器学习模型第3版以及扩展逻辑回归妊娠期糖尿病模型第3版(后两者各含8个分类与连续变量)。
- 模型性能通过区分度和校准评估,并采用决策曲线分析确定净获益,考虑重新校准以改善模型性能。
主要发现
- 模型第2版、第3版及新时间验证数据集的开发数据中,年龄≥35岁的孕妇分别占21.2%、22.5%和33.5%。
- 上述三个数据集中,体重指数≥30 kg/m2者分别占22%、23.7%和24.0%。
- 上述三个数据集中,妊娠期糖尿病患病率分别为18%、21.3%和28.6%。
- 各模型区分度相似,第2版、第3版机器学习模型和第3版逻辑回归模型的AUC分别为0.72(95% CI:0.71,0.73)、0.73(95% CI:0.72,0.74)和0.73(95% CI:0.73,0.74)。
- 所有模型均出现过估计,校准斜率分别为0.87、0.99和0.87,重新校准后改善。
局限性
- 对于所有模型,在不同种族和产次中观察到预测性能存在一些变异性。
- 原始机器学习模型在验证期间性能显著下降。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: This study aimed to follow best practice by temporally evaluating existing gestational diabetes mellitus (GDM) prediction models, updating them where needed, and comparing the temporal evaluation performance of the machine learning (ML)-based models with that of regression-based models. STUDY DESIGN AND SETTING: We utilized new data for the temporal validation dataset with 12,722 singleton pregnancies at the Monash Health Network from 2021 to 2022. The Monash GDM Logistic Regression (LR) model with six categorical variables (version 2) and the Monash GDM ML model (version 3), along with an extended LR GDM model (version 3), each with eight categorical and continuous variables, were evaluated. Model performance was assessed using discrimination and calibration. Decision curve analyses (DCA) were performed to determine the net benefit of models. Recalibration was considered to improve model performance. RESULTS: The development datasets for model versions 2, 3, and the new temporal validation dataset included 21.2%, 22.5%, and 33.5% of pregnant women aged ≥35 years, respectively; 22%, 23.7%, and 24.0% with a body mass index ≥30 kg/m2; and GDM prevalence rates of 18%, 21.3%, and 28.6%, respectively. There was similar discrimination performance across the models, with area under the receiver operating characteristic curve (AUC) of 0.72 [95% CI: 0.71, 0.73], 0.73 [95% CI: 0.72, 0.74], and 0.73 [95% CI: 0.73, 0.74] for version 2 and version 3 ML and LR models, respectively. All models exhibited overestimation with calibration slopes of 0.87, 0.99, and 0.87, respectively, which improved with recalibration. DCA showed that all models had better net benefits as compared to treat all and treat none. For all models, some variability has been observed in prediction performance across ethnic groups and parity. CONCLUSION: Despite significant changes in the background characteristics of the population, we have demonstrated that all models remained robust, especially after recalibration. However, the performance of the original ML model decreased significantly during validation. Dynamic models are better suited to adapt to the temporal changes in baseline characteristics of pregnant women and the resulting calibration drift, as they can incorporate new data without requiring manual evaluation.
引用
- Belsti Y, Moran L, Mousa A, et al. Evaluation of machine learning and logistic regression-based gestational diabetes prognostic models. J Clin Epidemiol. 2025. PMID: 40886901. doi:10.1016/j.jclinepi.2025.111957
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。