使用响应置换衡量影像数据集对所选标准分析流程的过拟合易感性
该研究提出一种量化过拟合对受试者工作特征曲线下面积(AUC)影响的方法,在乳腺癌诊断的公开影像与病理数据集上,通过50次结局置换并比较七类分类模型与四种评估技术,发现未控制过拟合时AUC可接近1.0,交叉验证防护优于其他技术,多数算法需至少200例样本才能以小于0.05的AUC膨胀评估少至10个特征。
要点
- 未控制过拟合时,影像组学模型AUC可接近1.0。
- 交叉验证对过拟合的防护优于其他评估技术。
- 多数分类算法评估10个特征需至少200例样本。
核心结论
- 该方法可应用于任何精选数据集,以提示限制过拟合所需的特征数量和分析方法。
研究设计与样本
- 方法学研究,使用两个公开的乳腺癌诊断影像与病理图像数据库。
- 对每个数据集置换结局(癌症诊断)值以消除影像特征与结局的真实关联。
- 七类分类模型(逻辑回归、线性判别分析、朴素贝叶斯、线性支持向量机、非线性支持向量机、随机森林、多层感知机)拟合至打乱数据集。
- 四种评估技术(全部数据、留出法、10折交叉验证、自助法)用于评估,共重复50次结局置换后取AUC平均值。
诊断标准与关键数值
- 以零AUC 0.5为基准,AUC超出0.5的部分归因于过拟合。
- 未控制过拟合时可出现接近完美的预测(AUC接近1.0)。
- 交叉验证对过拟合的防护优于其他评估技术。
- 多数分类算法评估少至10个特征需至少200例样本,才能使过拟合导致的AUC膨胀小于0.05。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: This study demonstrates a method for quantifying the impact of overfitting on the receiving operator characteristic curve (AUC) when using standard analysis pipelines to develop imaging biomarkers. We illustrate the approach using two publicly available repositories of radiology and pathology images for breast cancer diagnosis.
MATERIALS AND METHODS: For each dataset, we permuted the outcome (cancer diagnosis) values to eliminate any true association between imaging features and outcome. Seven types of classification models (logistic regression, linear discriminant analysis, Naïve Bayes, linear support vector machines, nonlinear support vector machine, random forest, and multi-layer perceptron) were fitted to each scrambled dataset and evaluated by each of four techniques (all data, hold-out, 10-fold cross-validation, and bootstrapping). After repeating this process for a total of 50 outcome permutations, we averaged the resulting AUCs. Any increase over a null AUC of 0.5 can be attributed to overfitting.
RESULTS: Applying this approach and varying sample size and the number of imaging features, we found that failing to control for overfitting could result in near-perfect prediction (AUC near 1.0). Cross-validation offered greater protection against overfitting than the other evaluation techniques, and for most classification algorithms a sample size of at least 200 was required to assess as few as 10 features with less than 0.05 AUC inflation attributable to overfitting.
CONCLUSION: This approach could be applied to any curated dataset to suggest the number of features and analysis approaches to limit overfitting.
引用
- Chakraborty J, Midya A, Kurland BF, et al. Use of Response Permutation to Measure an Imaging Dataset's Susceptibility to Overfitting by Selected Standard Analysis Pipelines. Acad Radiol. 2024. PMID: 38614825. doi:10.1016/j.acra.2024.02.028
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。