受试者工作特征曲线在不平衡数据中的局限性:辅助装置死亡风险评分
在左心室辅助装置领域,受试者工作特征曲线因幸存者占绝大多数(不平衡数据)而可能扭曲对短期死亡预测能力的评估。本研究以800例连续流左心室辅助装置患者为对象,比较HeartMate风险评分与随机森林两种分类器对90天死亡的预测,并引入精确率-召回率曲线作为补充指标。结果显示受试者工作特征曲线下面积分别为0.77和0.63,而精确率-召回率曲线下面积分别为0.43和0.16,提示后者更能反映少数类预测性能。
要点
- 在不平衡数据中,受试者工作特征曲线可能高估分类器性能。
- 精确率-召回率曲线更关注少数类,可补充评估死亡预测。
- HeartMate风险评分的精确率随敏感度轻微升高迅速降至10%。
核心结论
- 受试者工作特征曲线在不平衡数据中可能过于乐观地呈现分类器或风险评分的性能。
- 精确率-召回率曲线通过聚焦少数类,能更好反映分类器预测左心室辅助装置死亡的性能。
研究设计与样本
- 研究比较两种分类器(HeartMate风险评分与随机森林)对90天左心室辅助装置死亡的预测。
- 测试组为Interagency Registry for Mechanically Assisted Circulatory Support登记的800例2006至2016年接受连续流左心室辅助装置的患者(平均年龄59岁;146例女性 vs 654例男性),90天死亡率为8%。
主要发现
- 受试者工作特征曲线下面积:随机森林0.77,HeartMate风险评分0.63。
- 精确率-召回率曲线下面积:随机森林0.43,HeartMate风险评分0.16。
- HeartMate风险评分的精确率随敏感度轻微升高迅速降至10%。
临床怎么用
- 在心外重症与机械循环支持领域评估死亡预测模型时,应意识到受试者工作特征曲线在不平衡数据中的局限性。
- 可补充使用精确率-召回率曲线来更全面评价分类器对少数类(死亡)的预测性能。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: In the left ventricular assist device domain, the receiver operating characteristic is a commonly applied metric of performance of classifiers. However, the receiver operating characteristic can provide a distorted view of classifiers' ability to predict short-term mortality due to the overwhelmingly greater proportion of patients who survive, that is, imbalanced data. This study illustrates the ambiguity of the receiver operating characteristic in evaluating 2 classifiers of 90-day left ventricular assist device mortality and introduces the precision recall curve as a supplemental metric that is more representative of left ventricular assist device classifiers in predicting the minority class. METHODS: This study compared the receiver operating characteristic and precision recall curve for 2 classifiers for 90-day left ventricular assist device mortality, HeartMate Risk Score and Random Forest for 800 patients (test group) recorded in the Interagency Registry for Mechanically Assisted Circulatory Support who received a continuous-flow left ventricular assist device between 2006 and 2016 (mean age, 59 years; 146 female vs 654 male patients), in whom 90-day mortality rate is only 8%. RESULTS: The receiver operating characteristic indicates similar performance of Random Forest and HeartMate Risk Score classifiers with respect to area under the curve of 0.77 and Random Forest 0.63, respectively. This is in contrast to their precision recall curve with area under the curve of 0.43 versus 0.16 for Random Forest and HeartMate Risk Score, respectively. The precision recall curve for HeartMate Risk Score showed the precision rapidly decreased to only 10% with slightly increasing sensitivity. CONCLUSIONS: The receiver operating characteristic can portray an overly optimistic performance of a classifier or risk score when applied to imbalanced data. The precision recall curve provides better insight about the performance of a classifier by focusing on the minority class.
引用
- Movahedi F, Padman R, Antaki JF, et al. Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores. J Thorac Cardiovasc Surg. 2021. PMID: 34446286. doi:10.1016/j.jtcvs.2021.07.041
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。