临床机器学习模型回顾性评价的连续评估框架
该文针对临床机器学习模型通常仅在单一时间点用AUROC、AUPRC等指标评估的局限,提出三种回顾性连续评估框架:分区法、预测活动监测特征曲线(AMOC)法和通知法。作者将三种框架应用于两个公开数据集上的脓毒症预测模型,结果显示高AUROC的模型在连续评估时表现出显著不同的性能。三种框架提供了超越单时间点指标的多维度模型性能信息,有助于更真实地理解模型临床效用,并支持模型选择、部署和监测。
要点
- 单时间点AUROC、AUPRC主要评估区分度,不能反映模型连续运行时的行为和临床效用。
- 作者提出分区法、预测活动监测特征曲线(AMOC)法和通知法三种连续评估框架,用于回顾性数据集上的临床机器学习模型评价。
- 在两个公开数据集上,高AUROC的脓毒症预测模型在连续评估时性能差异明显。
- 三种框架揭示单时间点指标未覆盖的时间维度和行为维度,帮助更真实地理解模型效用。
- 连续评估可支持模型选择、部署和监测,提升机器学习系统的信任度、透明度和临床适用性。
适用人群与场景
- 研究对象为两个公开临床数据集上的脓毒症预测模型,评估场景为回顾性数据集上的连续模型评价。
研究设计与关键结果
- 研究类型为方法学框架开发与回顾性评估,应用三种连续评估框架(分区法、预测活动监测特征曲线法、通知法)于两个公开数据集上的脓毒症预测模型。
- 在两个数据集上,高AUROC的模型在连续评估时表现出显著不同的性能。
- 三种框架揭示传统单时间点指标未捕获的多维度模型性能,包括时间维度和行为维度。
证据类型、等级与局限
- 证据类型为方法学研究,基于两个公开数据集的回顾性评估,原文未报告证据等级或推荐强度。
- 局限:原文未报告局限性。
对床旁实践的改变
- 作者认为,所提出的分区法、预测活动监测特征曲线法和通知法框架能更全面、更真实地评估连续运行的临床机器学习模型。
- 作者认为,这些方法通过揭示模型性能的时间维度和行为维度,有助于在医疗环境中进行更明智的模型选择、部署和监测。
- 作者认为,实施连续评估可提升机器学习系统的信任度、透明度和临床适用性,超越单时间点指标所能提供的信息。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
The performance of clinical machine learning (ML) models that continuously predict the risk of a future event is typically evaluated using metrics at a single time point. While these metrics are widely accepted to evaluate the performance of ML models, they are not comprehensive enough to capture the performance and behavior of the ML models when executing continuously in real-world settings. Additionally, these single time point metrics assess model discriminability, which is different from clinical utility. In this paper, we propose three novel frameworks to continuously evaluate clinical ML models on retrospective datasets: (1) Zone-based (2) Predictive Activity Monitoring Characteristic curve-based and (3) Notification-based. We demonstrate the value of these three continuous evaluation frameworks using predictions from sepsis models evaluated on two publicly available datasets. The three proposed frameworks provide a more comprehensive evaluation and comparison of different sepsis prediction models and reveal additional aspects of model performance beyond standard metrics computed at a single time point. These frameworks provide users of clinical ML models a more realistic understanding of the model’s utility. BACKGROUND: Clinical machine learning (ML) models are increasingly used to predict patient risk continuously for future adverse events, such as sepsis. Traditionally, these models are evaluated using performance metrics computed at a single timepoint. While these metrics—such as Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision Recall curve ( AUPRC)—are widely accepted, they fail to capture how a model behaves when operating continuously in real-world clinical environments. Moreover, single time point evaluations primarily assess discriminability, which may not directly reflect a model’s clinical utility. METHODS: We propose and implement three novel retrospective evaluation frameworks to continuously assess clinical ML models: zone-based, predictive activity monitoring characteristic curve (AMOC)-based and notification-based. These frameworks were applied to sepsis prediction models using two publicly available clinical datasets. Model outputs were continuously monitored, and results were compared against traditional single time point metrics to assess differences in performance. RESULTS: The three continuous evaluation frameworks reveal multiple dimensions of model performance not captured by conventional metrics. Across both datasets, models with high AUROC demonstrate markedly different performance when continuously evaluated. CONCLUSIONS: The proposed zone-based, predictive AMOC-based, and notification-based frameworks offer a more comprehensive and realistic assessment of continuously operating clinical ML models. By uncovering temporal and behavioral aspects of model performance, these methods enable more informed model selection, deployment, and monitoring in healthcare settings. Implementing continuous evaluation can enhance trust, transparency, and clinical applicability of ML systems beyond what single time point metrics provide. TRIAL REGISTRATION: Not Applicable
参考来源
- Natarajan A, Chang Y, McLaverty B, et al. Continuous Evaluation Frameworks for Retrospective Evaluation of Clinical Machine Learning Models. Crit Care. 2026. PMID: 41749266. doi:10.1186/s13054-025-05725-9
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。