预测模型内部验证方法的实证模拟:比较k折交叉验证与基于bootstrap的乐观校正
该研究基于MIMIC-IV数据库的239,415例住院患者数据,通过模拟比较k折交叉验证与基于bootstrap的乐观校正方法在统计和机器学习模型内部验证中的表现。结果显示,对于参数模型,.632+估计量最准确,10折交叉验证仅有轻微偏倚;对于非参数模型,所有bootstrap校正方法均显著高估AUC且不随样本量增加而改善,而10折交叉验证在所有场景下表现稳定良好。作者推荐优先将10折交叉验证作为预测模型的内部验证方法。
要点
- 参数模型中.632+估计量AUC最准确,10折交叉验证仅轻微偏倚。
- 非参数模型中bootstrap校正方法显著高估AUC,增加样本量未减少高估。
- 10折交叉验证在所有建模策略和样本量场景下表现稳定良好。
核心结论
- 基于bootstrap的乐观校正方法性能受模型复杂度影响,.632+估计量在基于小样本训练的参数模型中表现最佳。
- 10折交叉验证更稳健且更易实施,推荐优先作为预测模型的内部验证方法。
研究设计与样本
- 从开放获取数据库MIMIC-IV提取239,415例住院患者,随机抽取39,145例作为预定义参考数据集。
- 在剩余200,000例模拟数据集中随机选取样本量595至5946的训练集,采用逻辑回归、LASSO回归、朴素贝叶斯、支持向量机、K近邻、LightGBM和随机森林等多种建模策略建立预测模型。
- 模型因变量为急性肾损伤(AKI),发生率为18.5%,自变量包括22个常见AKI预测因子。
- 对每个模型采用2折、5折、10折交叉验证及Harrell、.632、.632+ bootstrap AUC估计量进行内部验证,模拟过程重复1000次获得1000个AUC估计值,并与参考数据集中的经验AUC比较。
主要发现
- 对于参数模型,.632+估计量提供最准确的AUC估计,10折交叉验证仅有轻微偏倚。
- 对于非参数模型,所有基于bootstrap的乐观校正方法均显著高估AUC,且高估未随样本量增加而减少。
- 10折交叉验证在所有考虑的场景中表现稳定且良好,不受建模策略或样本量影响。
临床怎么用
- 作者建议优先将10折交叉验证作为预测模型的内部验证方法。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND AND OBJECTIVE: To systematically evaluate the performance of k-fold cross-validation and bootstrap-based optimism correction methods for internal validation of statistical and machine learning models. METHODS: A total of 239,415 inpatients were extracted from an open access database named Medical Information Mart for Intensive Care IV, of which 39,145 were randomly sampled as a predefined reference dataset. Among the remaining simulation dataset with 200,000 inpatients, training sets with sample sizes ranging from 595 to 5946 were randomly selected, and multiple prediction models were developed in each training set using various modeling strategies, including logistic regression, least absolute shrinkage and selection operator regression, Naive Bayes, Support Vector Machine, K-Nearest Neighbors, Light Gradient Boosting Machine, and Random Forest. The dependent variable of the model was acute kidney injury (AKI), a binary outcome with an incidence of 18.5%, and the independent variables included 22 common predictors of AKI. For each model, 2-fold, 5-fold, and 10-fold cross-validation were used for internal validation to calculate area under the receiver-operating characteristic curve (AUC), which is a common metric for quantifying the overall ability of a model to discriminate between positive or negative classifications. In addition, the Harrell, .632, and .632+ AUC estimators were calculated for internal validation based on bootstrapping. The above simulation process was repeated 1000 times to obtain 1000 estimates of AUC for each internal validation method of each model. The model performance was simultaneously evaluated in the reference dataset to obtain an empirical AUC (analogous to the "gold standard"). Then, by comparing the 1000 AUC estimates with the empirical AUC, the accuracy of internal validation methods for different models was assessed. RESULTS: For parametric models, the .632+ estimator provided the most accurate estimates of AUC, followed by 10-fold cross-validation with only slight bias. In contrast, for nonparametric models, all bootstrap-based optimism correction methods significantly overestimated AUC, and the overestimation was not reduced by increasing the sample size. Most strikingly, 10-fold cross-validation demonstrated stable and good performance across all scenarios considered, regardless of the modeling strategy or sample size. CONCLUSION: The performance of bootstrap-based optimism correction methods can be affected by model complexity, although the .632+ estimator performs best in parameter models based on small-sample training. In comparison, 10-fold cross-validation is more robust and easier to implement. Therefore, it is recommended to prioritize 10-fold cross-validation as the internal validation method for prediction models. PLAIN LANGUAGE SUMMARY: With the exponential growth of clinical prediction models, the methods for conducting internal validation of these models remain controversial. Both k-fold cross-validation and bootstrap-based optimism correction methods are recommended by guidance papers. However, the issue of whether they are applicable to all modeling strategies, especially machine learning algorithms, still lacks evidence. This study simulated various sample size scenarios based on real-world clinical data, and developed AKI prediction models based on parametric and nonparametric modeling strategies. Then, internal validation was performed for each model using different methods. The results showed that bootstrap-based optimism correction methods were suitable for parametric models. However, as the model complexity increased, the bias of bootstrap-based optimism correction methods increased accordingly. In contrast, 10-fold cross-validation performed well in all scenarios, regardless of the modeling strategy or sample size. Therefore, 10-fold cross-validation is recommended as a preferred method for internal validation of prediction models.
引用
- Zhang C, Yan R, Liu X, et al. Empirical simulation of internal validation methods for prediction models: comparing k-fold cross-validation with bootstrap-based optimism correction. J Clin Epidemiol. 2025. PMID: 41391734. doi:10.1016/j.jclinepi.2025.112101
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。