多机构数据集对ICU机器学习预测模型泛化能力的影响
该回顾性观察性队列研究利用四个公共数据集整合的334,812例ICU住院数据,系统评估深度学习模型对死亡、急性肾损伤(AKI)和脓毒症三种不良事件的跨院迁移能力。模型在训练医院的AUROC分别为死亡0.838-0.869、AKI 0.823-0.866、脓毒症0.749-0.824,但在其他医院应用时AUROC下降,有时降幅达-0.200。使用多个数据集训练可减轻性能下降,多中心模型表现与最佳单中心模型大致相当;专门促进泛化的方法在本实验中未见明显改善。作者认为,多样化的训练数据对基于深度学习的风险预测很重要,随着更多医院数据可用于训练,模型可能越来越具有泛化性,但在新医院的良好表现仍取决于训练时纳入了兼容的医院。
要点
- 在训练医院,深度学习模型对死亡的AUROC为0.838-0.869,对AKI为0.823-0.866,对脓毒症为0.749-0.824。
- 模型应用到其他医院时AUROC下降,有时降幅达-0.200。
- 使用多个数据集训练可减轻性能下降,多中心模型表现与最佳单中心模型大致相当。
- 专门促进泛化的方法在本实验中未见明显改善模型在新医院的表现。
- 作者认为,新医院的良好表现仍取决于训练时纳入了兼容的医院。
适用人群与场景
- 入住ICU至少6小时且数据质量良好的成人患者。
研究设计与关键结果
- 研究类型为回顾性观察性队列研究,使用四个公共数据集整合的ICU数据,共334,812例ICU住院,场景为欧洲和美国的ICU。
- 评估深度学习模型对死亡、急性肾损伤(AKI)和脓毒症三种常见不良事件的跨院迁移能力。
- 在训练医院,模型对死亡的AUROC为0.838-0.869,对AKI为0.823-0.866,对脓毒症为0.749-0.824。
- 模型应用到其他医院时AUROC下降,有时降幅达-0.200。
- 使用多个数据集训练可减轻性能下降,多中心模型表现与最佳单中心模型大致相当。
- 专门促进泛化的方法在本实验中未见明显改善模型在新医院的表现。
证据类型、等级与局限
- 证据类型:回顾性观察性队列研究。
- 局限:原文未报告局限性。
对床旁实践的改变
- 作者认为,多样化的训练数据对基于深度学习的风险预测很重要。
- 作者认为,随着更多医院数据可用于训练,模型可能越来越具有泛化性;但在新医院的良好表现仍取决于训练时纳入了兼容的医院。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
OBJECTIVES: To evaluate the transferability of deep learning (DL) models for the early detection of adverse events to previously unseen hospitals.
DESIGN: Retrospective observational cohort study utilizing harmonized intensive care data from four public datasets.
SETTING: ICUs across Europe and the United States.
PATIENTS: Adult patients admitted to the ICU for at least 6 hours who had good data quality.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: Using carefully harmonized data from a total of 334,812 ICU stays, we systematically assessed the transferability of DL models for three common adverse events: death, acute kidney injury (AKI), and sepsis. We tested whether using more than one data source and/or algorithmically optimizing for generalizability during training improves model performance at new hospitals. We found that models achieved high area under the receiver operating characteristic (AUROC) for mortality (0.838-0.869), AKI (0.823-0.866), and sepsis (0.749-0.824) at the training hospital. As expected, AUROC dropped when models were applied at other hospitals, sometimes by as much as -0.200. Using more than one dataset for training mitigated the performance drop, with multicenter models performing roughly on par with the best single-center model. Dedicated methods promoting generalizability did not noticeably improve performance in our experiments.
CONCLUSIONS: Our results emphasize the importance of diverse training data for DL-based risk prediction. They suggest that as data from more hospitals become available for training, models may become increasingly generalizable. Even so, good performance at a new hospital still depended on the inclusion of compatible hospitals during training.
参考来源
- Rockenschaub P, Hilbert A, Kossen T, et al. The Impact of Multi-Institution Datasets on the Generalizability of Machine Learning Prediction Models in the ICU. Crit Care Med. 2024. PMID: 38958568. doi:10.1097/CCM.0000000000006359
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。