合成影像组学数据质量指标无法预测外部验证增益
本研究在50个公开影像组学二分类数据集上训练三种条件生成器(WGAN-GP、CVAE、TabDDPM),评估五种合成数据质量指标与模拟外部验证中ΔAUC的关联。汇总分析显示质量指标与ΔAUC显著相关(|ρ|=0.30–0.36),但生成器内亚组分析全部不显著,呈现生态学谬误。质量导向的生成器选择反而产生显著负向汇总Δ(−0.0051,95%CI −0.0083至−0.0019),提示质量指标不能替代任务特异的外部验证。
要点
- 合成数据质量指标无法判断增强是否改善模型外部验证表现,不能作为临床效用的代理指标。
- 汇总层面的质量-性能相关性由生成器间差异驱动,生成器内部无预测价值,存在生态学谬误。
- 部署前必须进行任务特异的外部验证,质量指标不能替代这一环节。
核心结论
- 合成数据质量指标与性能增益的汇总相关性源于生成器间差异,生成器内不具预测价值。
- 质量导向生成器选择相较于仅真实数据训练表现更差,汇总Δ为−0.0051(95%CI:−0.0083至−0.0019)。
- 随机森林复现实验中汇总相关性全部不显著(所有p>0.09),核心结论不依赖于分类器。
研究设计与样本
- 基于50个radMLBench公开影像组学二分类数据集,训练WGAN-GP、CVAE和TabDDPM三种条件生成器。
- 共150个数据集-生成器配对,记录五种质量指标和模拟域偏移下的ΔAUC,重复10次。
- 单类测试折导致的未定义AUC在外部验证中排除30/2500例(1.2%)。
- 使用随机森林分类器独立复现全部实验,并预设对29/50个与机会水平可区分的数据集进行敏感性分析。
诊断标准与关键数值
- 汇总质量指标与ΔAUC的秩相关绝对值为0.30–0.36,校正后均p<0.001。
- 生成器内所有质量-性能相关性均不显著(均p>0.05),经Benjamini-Hochberg校正后仅5/20个相关存活。
- 质量指标对性能增益的ROC-AUC范围为0.42至0.61,接近机会水平。
- 质量导向选择仅在17/50(34%)数据集中改善外部AUC,而理想选择(oracle)为34/50(68%)。
临床应用与阅片要点
- 日常阅片与报告中不应以合成数据质量指标作为判断模型泛化能力的依据。
- 评估生成增强影像组学模型时,需坚持任务特异的模拟外部验证,而非依赖质量评分或距离度量。
- 同一生成器架构下质量指标变异缺乏决策指导力,不能用于选择增强策略或替代真实数据训练。
局限性
- 生成器内亚组分析的样本量受限于数据集数量,可能存在统计效能不足。
- 仅使用二分类影像组学数据集与三种条件生成器,结论向其他任务或生成架构的外推受限。
- 模拟外部验证的域偏移设计与真实临床部署中的数据分布变化可能不完全一致。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: To investigate whether established synthetic data quality metrics predict when deep generative augmentation improves performance under simulated external validation in radiomics.
MATERIALS AND METHODS: Three conditional generators (WGAN-GP, CVAE, TabDDPM) were trained on 50 public binary-classification radiomic datasets from radMLBench. For each dataset-generator pair (n = 150), five quality metrics and ΔAUC (change in simulated external AUC under domain shift) were recorded across 10 repetitions; undefined AUCs (single-class test folds) were excluded a priori (30/2500 external, 1.2%). Six analyses tested quality-ΔAUC associations: rank correlations with FDR correction, ROC discrimination, subgroup random-effects meta-analysis, composite scoring, quality-guided generator selection, and SMOTE comparison. All experiments were replicated with a random forest classifier.
RESULTS: Pooled quality metrics correlated significantly with ΔAUC (|ρ| = 0.30-0.36, all adjusted p < 0.001); within each generator, all correlations were non-significant (all p > 0.05), revealing an ecological fallacy. After Benjamini-Hochberg correction, 5 of 20 correlations survived: the 4 pooled associations and DDPM MMD (unexpected direction). ROC-AUC ranged from 0.42 to 0.61 (near-chance). Quality-guided selection yielded a significantly negative pooled Δ (-0.0051; 95% CI: -0.0083 to -0.0019) and improved external AUC in only 17/50 (34%) datasets versus 34/50 (68%) for an oracle. Random forest replication rendered pooled correlations non-significant (all p > 0.09). A pre-specified sensitivity analysis on the 29/50 datasets statistically distinguishable from chance produced identical conclusions.
CONCLUSION: Aggregate quality-performance correlations are driven by between-generator differences, not within-generator variation that could guide practical decisions. Quality metrics are insufficient proxies for clinical utility; task-specific external validation remains indispensable.
KEY POINTS: Question Can established quality metrics of synthetic radiomics data identify when generative augmentation actually improves the performance of prediction models under simulated external validation? Findings Across 50 public datasets, aggregate quality-performance correlations vanished within individual generators (ecological fallacy), and quality-guided generator selection underperformed real-only training. Clinical relevance Quality metrics commonly used to validate synthetic radiomics data cannot identify when augmentation improves clinical model performance under simulated domain shift, underscoring the irreplaceable role of task-specific external validation before clinical deployment.
引用
- García-Hidalgo, Consentino Hernández, Cayuela Espí, et al. Quality metrics of synthetic radiomics data do not predict improvement under simulated external validation: an ecological fallacy across 50 public datasets. Eur Radiol. 2026. PMID: 42321417. doi:10.1007/s00330-026-12703-4
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。