稀有事件下预测性能指标的表现
该模拟研究基于精神健康研究网络数据(149个预测变量,自杀未遂事件率0.92%),通过改变训练与评估数据集大小,探讨AUC及其他预测性能指标在稀有事件下的偏倚与方差来源。结果显示AUC表现由事件数而非事件率驱动,事件数达1000时偏倚接近零。
要点
- AUC在稀有事件下的偏倚与方差主要由事件数驱动,而非事件率。
- 模拟中事件数达1000时,AUC偏倚接近零。
- 阳性预测值和准确率即使在大样本中也依赖事件率。
核心结论
- AUC在稀有事件下可靠的前提是事件总数中等偏大;模拟中事件数达1000时观察到接近零的偏倚。
研究设计与样本
- 基于精神健康研究网络数据的plasmode模拟研究,数据含149个预测变量,结局为自杀未遂,原始数据集事件率0.92%。
主要发现
- AUC表现不佳(经验偏倚、交叉验证AUC估计的变异性及置信区间经验覆盖率)由稀有事件下的事件数驱动,而非事件率。
- 敏感性的表现由事件数驱动,特异性的表现由非事件数驱动。
- 阳性预测值和准确率等其他指标即使在大样本中也依赖事件率。
关键数值与推荐
- 模拟中事件数达1000时,AUC偏倚接近零。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: Area under the receiver operating characteristic curve (AUC) is commonly reported alongside prediction models for binary outcomes. Recent articles have raised concerns that AUC might be a misleading measure of prediction performance in the rare event setting. This setting is common since many events of clinical importance are rare. We aimed to determine whether the bias and variance of AUC are driven by the number of events or the event rate. We also investigated the behavior of other commonly used measures of prediction performance, including positive predictive value, accuracy, sensitivity, and specificity. STUDY DESIGN AND SETTING: We conducted a simulation study to determine when or whether AUC is unstable in the rare event setting by varying the size of datasets used to train and evaluate prediction models. This plasmode simulation study was based on data from the Mental Health Research Network; the data contained 149 predictors and the outcome of interest, suicide attempt, which had event rate 0.92% in the original dataset. RESULTS: Our results indicate that poor AUC behavior-as measured by empirical bias, variability of cross-validated AUC estimates, and empirical coverage of confidence intervals-is driven by the number of events in a rare-event setting, not event rate. Performance of sensitivity is driven by the number of events, while that of specificity is driven by the number of nonevents. Other measures, including positive predictive value and accuracy, depend on the event rate even in large samples. CONCLUSION: AUC is reliable in the rare event setting provided that the total number of events is moderately large; in our simulations, we observed near zero bias with 1000 events. PLAIN LANGUAGE SUMMARY: Predicting self-harm or suicidal behavior is medically important for guiding clinicians in providing care to patients. Several research teams have developed and evaluated suicide risk prediction models based on health records data. Part of evaluating these models is calculating area under the receiver operating characteristic curve (AUC) and other prediction performance metrics. Self-harm and suicide are rare events. Recent research has raised concerns with using AUC in rare-event settings. We aimed to determine whether having a sufficiently large dataset could remove these concerns. In our experiments, we found that AUC can be used without concern in settings with 1000 events or more. Thus, AUC is a valid measure of suicide risk prediction model performance in many large healthcare databases.
引用
- Minus E, Coley RY, Shortreed SM, et al. Behavior of prediction performance metrics with rare events. J Clin Epidemiol. 2025. PMID: 41224138. doi:10.1016/j.jclinepi.2025.112046
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。