机器学习与统计模型在预测个体心血管疾病风险中的一致性:纵向队列研究
结果显示各模型人群水平表现相似(C统计量约0.87),但个体风险预测差异很大,尤其在高风险患者中;忽略删失的模型会显著低估风险。建议临床决策前常规评估模型间及模型内一致性。
要点
- 不同模型人群水平C统计量约0.87且校准相似,但个体风险预测差异大。
- QRISK3预测9.5-10.5%的患者,随机森林和神经网络预测范围宽。
- 忽略删失的模型会显著低估心血管疾病风险,应优先用考虑删失的生存模型。
核心结论
- 多种模型对同一患者的心血管疾病风险预测差异很大,尽管模型性能相似。
- 应考虑删失的、可解释的生存模型如QRISK3更可取。
- 临床决策前应常规评估模型内和模型间的一致性。
研究设计与样本
- 应用19种预测技术:12类机器学习模型(网格搜索最佳模型)、3种Cox比例风险模型(局部拟合、QRISK3、Framingham)、3种参数生存模型和1种逻辑回归模型。
主要发现
- 各模型人群水平性能相似,C统计量约0.87且校准相似。
- 个体风险预测在不同类型机器学习与统计模型之间及内部差异很大,尤其在高风险患者中。
- QRISK3预测风险9.5-10.5%的患者,在随机森林中风险为2.9-9.2%,在神经网络中为2.4-7.2%。
- QRISK3与神经网络预测风险差异范围(95%范围)为-23.2%至0.1%。
关键数值与推荐
- 模型性能:C统计量约0.87,校准相似。
- QRISK3预测风险9.5-10.5%的患者,随机森林预测2.9-9.2%,神经网络预测2.4-7.2%。
- QRISK3与神经网络预测风险差异95%范围为-23.2%至0.1%。
临床怎么用
- 临床决策中不应直接应用未考虑删失的逻辑回归或常用机器学习模型预测长期风险。
- 优先选择考虑删失且可解释的生存模型如QRISK3。
- 使用前应常规评估模型内和模型间预测一致性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: To assess the consistency of machine learning and statistical techniques in predicting individual level and population level risks of cardiovascular disease and the effects of censoring on risk predictions. DESIGN: Longitudinal cohort study from 1 January 1998 to 31 December 2018. SETTING AND PARTICIPANTS: 3.6 million patients from the Clinical Practice Research Datalink registered at 391 general practices in England with linked hospital admission and mortality records. MAIN OUTCOME MEASURES: Model performance including discrimination, calibration, and consistency of individual risk prediction for the same patients among models with comparable model performance. 19 different prediction techniques were applied, including 12 families of machine learning models (grid searched for best models), three Cox proportional hazards models (local fitted, QRISK3, and Framingham), three parametric survival models, and one logistic model. RESULTS: The various models had similar population level performance (C statistics of about 0.87 and similar calibration). However, the predictions for individual risks of cardiovascular disease varied widely between and within different types of machine learning and statistical models, especially in patients with higher risks. A patient with a risk of 9.5-10.5% predicted by QRISK3 had a risk of 2.9-9.2% in a random forest and 2.4-7.2% in a neural network. The differences in predicted risks between QRISK3 and a neural network ranged between -23.2% and 0.1% (95% range). Models that ignored censoring (that is, assumed censored patients to be event free) substantially underestimated risk of cardiovascular disease. Of the 223 815 patients with a cardiovascular disease risk above 7.5% with QRISK3, 57.8% would be reclassified below 7.5% when using another model. CONCLUSIONS: A variety of models predicted risks for the same patients very differently despite similar model performances. The logistic models and commonly used machine learning models should not be directly applied to the prediction of long term risks without considering censoring. Survival models that consider censoring and that are explainable, such as QRISK3, are preferable. The level of consistency within and between models should be routinely assessed before they are used for clinical decision making.
引用
- Li Y, Sperrin M, Ashcroft DM, et al. Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar. BMJ. 2020. PMID: 33148619. doi:10.1136/bmj.m3919
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。