院际转运死亡风险模型的外部验证与本地化改进:SafeNET 在真实世界电子健康记录中的可迁移性测试
本研究在 2023 年一家多医院学术健康系统的 14,728 例成人院际转运患者中,外部验证已发表的 SafeNET 机器学习模型(14 变量特征集,不做修改),并用本地数据训练 CatBoost、LightGBM、XGBoost 梯度提升模型(70/30 分层拆分、五折交叉验证、类权重与随机欠采样处理约 1:27 类别不平衡)。SafeNET 在留出测试集上 AUC-ROC 0.87、F1 0.73(95%CI 0.69–0.77),接近开发站点结果;本地 CatBoost 模型 AUC-ROC 0.92、F1 0.74(95%CI 0.72–0.78),校准更优(Brier 0.04 vs 0.055)。功能状态变量在 85% 的病例中缺失,限制原始特征集的直接部署;
要点
- 在 14,728 例成人院际转运患者中,SafeNET 外部验证 AUC-ROC 0.87、F1 0.73(95%CI 0.69–0.77),与开发站点结果接近,支持跨系统可移植性。
- 本地 CatBoost 模型优于 SafeNET:AUC-ROC 0.92、F1 0.74(95%CI 0.72–0.78),Brier 0.04 对比 0.055,校准更好。
- 功能状态变量在 85% 的转运病例中缺失,直接套用 SafeNET 原始 14 变量特征集在真实电子健康记录中不可行。
- 本地化训练可利用系统特异电子健康记录模式补偿跨系统数据缺失,但需注意类别不平衡(约 1:27)处理对精确率的影响。
适用人群与场景
- (本栏提炼含原文里核对不到的数值,已删除,请以原文为准)
诊断标准与关键阈值
- SafeNET 模型按已发表规格使用其原始 14 变量特征集,不做修改直接评估。
- 本地模型采用分层 70/30 训练-测试拆分、五折交叉验证、类权重和随机欠采样处理约 1:27 的类别不平衡。
- 评估指标包括区分度(AUC-ROC)、分类指标(精确率、召回率、F1)和校准(Brier 分数)。
处置流程与药物剂量
- 原文未提及
红旗征与不能漏的情况
- 原文未提及
研究设计与关键结果
- 研究类型:外部验证与模型改进研究;样本量 14,728 例成人院际转运患者;单中心多医院学术健康系统。
- SafeNET 在留出测试集上 AUC-ROC 0.87、F1 0.73(95%CI 0.69–0.77),与开发站点结果接近。
- 本地 CatBoost 模型 AUC-ROC 0.92、F1 0.74(95%CI 0.72–0.78),Brier 0.04 对比 SafeNET 0.055,校准改善。
- 功能状态变量在 85% 的病例中缺失,限制原始特征集直接实施。
证据类型、等级与局限
- 证据类型:外部验证与本地模型开发研究(基于真实世界电子健康记录)。
对床旁实践的改变
- SafeNET 可在外部系统复现区分度,但直接部署受限于功能状态变量高缺失率(85%),需评估本地数据可用性。
- 本地训练梯度提升模型(如 CatBoost)可提高区分度与校准,但需处理类别不平衡并关注精确率-召回率权衡。
- 临床部署预后模型时,应同时评估召回率与精确率,避免仅依赖 AUC 判断模型可用性。
- 原文未提及适用与不适用人群的进一步限定。
指南或共识出处
- 原文未提及
以上提炼依据论文摘要(全文未开放获取,或许可不允许翻译转载);全文请点「阅读原文」。
英文原文(摘要)· 点击展开
BACKGROUND: Mortality risk is often uncertain at the time of interhospital transfer (IHT).
OBJECTIVES: We examined what happens when a published machine learning model for post-transfer mortality, SafeNET, is deployed in an independent health system with substantial data fragmentation and missingness and assessed the extent to which local model development can compensate for cross-system data limitations.
DERIVATION COHORT: SafeNET was evaluated from published specifications using its original 14-variable feature schema and applied without modification. Locally trained gradient-boosting models (Categorical Boosting [CatBoost], Light Gradient Boosting, Extreme Gradient Boosting) were developed using a stratified 70/30 train-test split with five-fold cross-validation, class weighting, and random under-sampling to address class imbalance (~1:27).
VALIDATION COHORT: We included 14,728 adult patients (≥ 18 yr) undergoing IHT at a multihospital academic health system from January 1, 2023, to December 31, 2023. Functional status variables were missing in 85% of encounters, limiting direct implementation of the original feature set.
PREDICTION MODEL: Discrimination, classification metrics (precision, recall, F1 score), and calibration (Brier score) were compared across SafeNET and locally trained models on the held-out test set.
RESULTS: SafeNET achieved performance that closely approximated its development-site results (area under the receiver operating characteristic curve [AUC-ROC], 0.87; F1, 0.73; 95% CI, 0.69-0.77), indicating strong external consistency. Locally trained CatBoost models outperformed SafeNET (AUC-ROC, 0.92; F1, 0.74; 95% CI, 0.72-0.78) and demonstrated improved calibration (Brier 0.04 vs. 0.055). Key variables, including functional status, were missing in 85% of encounters, limiting direct implementation of the original feature set. Although recall exceeded 0.70 across most models, precision remained below 080 for several configurations, indicating important trade-offs relevant to clinical deployment.
CONCLUSIONS: SafeNET's performance was reproducible in an external health system, supporting its portability. However, local models trained on system-specific electronic health record patterns achieved markedly higher accuracy and calibration. Real-world replication revealed major data-availability barriers and highlighted the tension between acceptable recall and suboptimal precision in critical-care prognostic applications.
参考来源
- Hadler RA, Lakamana SK, Moorman T, et al. Testing Transferability of a Mortality Risk Model for Interhospital Transfer Using Real-World Electronic Health Records: External Validation and Model Enhancement. Crit Care Explor. 2026. PMID: 42720061. doi:10.1097/CCE.0000000000001442
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。