文献速递骨肿瘤

偏倚还是最佳拟合?SEER与NCDB数据集在单模型机器学习预测骨肉瘤生存结局中的比较分析

Bias or Best Fit? A Comparative Analysis of the SEER and NCDB Data Sets in Single-model Machine Learning for Predicting Osteosarcoma Survival Outcomes.
PubMed 文献 · Clin Orthop Relat Res · 2025-09-23

本研究基于SEER(4278例)和NCDB(4049例)骨肉瘤患者数据分别构建机器学习模型,内部验证AUC均大于0.90,但交叉外部验证AUC降至0.56至0.67,表明模型学习的是数据库特异性模式而非可泛化的疾病规律,跨数据库应用前需重新验证。

要点

  • SEER与NCDB各自训练的模型内部验证AUC均大于0.90,判别能力优秀。
  • 交叉外部验证AUC仅0.56至0.67,跨数据库预测性能差,不适用于临床决策。
  • NCDB模型侧重治疗变量,SEER模型侧重人口学因素,反映数据库信息差异。

核心结论

  • 模型应在与其应用相同的数据库环境中进行验证,跨数据库应用会导致预测性能差,未经重新验证应避免跨数据库使用。

研究设计与样本

  • 回顾性分析SEER(2000至2018年)和NCDB(2004至2018年)骨肉瘤患者数据,最终纳入SEER 4278例、NCDB 4049例,分别按80%训练集和20%验证集随机拆分构建机器学习模型。
  • SEER提供基于人群的覆盖和详细分期但治疗信息有限,NCDB提供基于医院的数据和全面治疗细节。
  • 证据等级为III级,预后研究。

主要发现

  • 内部验证:NCDB模型2年AUC为0.93(95%CI 0.92至0.94)、5年AUC为0.91(95%CI 0.90至0.92);SEER模型2年AUC为0.90(95%CI 0.89至0.91)、5年AUC为0.92(95%CI 0.91至0.92)。
  • 外部验证:NCDB模型在SEER数据上2年AUC为0.67(95%CI 0.65至0.68)、5年AUC为0.60(95%CI 0.58至0.62);SEER模型在NCDB数据上2年AUC为0.61(95%CI 0.59至0.62)、5年AUC为0.56(95%CI 0.55至0.58)。
  • NCDB模型优先使用治疗变量,SEER模型强调人口学因素,反映各数据库捕获的临床信息不同。
  • NCDB患者中位年龄22岁 vs SEER 17岁(p=0.005),性别分布相似(NCDB男性56%,SEER男性56%),种族构成和总生存率不同(NCDB 2年和5年分别为72%和52%,SEER分别为65%和43%)。

关键数值与推荐

  • AUC大于0.8提示判别能力良好,小于0.25的Brier评分提示模型有用。
  • 外部验证AUC小于0.70提示预测性能差(仅略优于随机,不适用于临床决策)。

临床怎么用

  • 原文结论认为模型应在与其应用相同的数据库环境中验证,跨数据库应用会导致预测性能差,未经重新验证应避免跨数据库使用。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Machine-learning models are increasingly used in orthopaedic oncology to predict survival outcomes for patients with osteosarcoma. Typically, these models are trained on a single data set, such as the Surveillance, Epidemiology, and End Results (SEER) or the National Cancer Database (NCDB). However, because any single database, even if it is large, may emphasize different data points and may include errors, models trained on single data sets may learn database-specific patterns rather than generalizable clinical relationships, limiting their clinical utility when applied to different patient populations. QUESTIONS/PURPOSES: We developed separate machine-learning models using SEER and NCDB databases and (1) compared the accuracy of SEER- and NCDB-trained models in estimating 2- and 5-year overall survival when validated on their respective databases, (2) assessed which database produced a more generalizable machine-learning model (defined as one that maintains high performance when applied to unseen external data) by using the model trained on one database to externally validate the other, and (3) identified key factors contributing to prediction accuracy. METHODS: From 2000 to 2018 (SEER) and 2004 to 2018 (NCDB), we identified 15,241 SEER patients and 11,643 NCDB patients with osteosarcoma. After excluding patients with tumors outside the extremities/pelvis, including unconfirmed osteosarcoma histology results (52% [7989] SEER, 22% [2537] NCDB) and those with missing metastasis, treatment, or prognosis data (20% [2974] SEER, 43% [5057] NCDB), we included 4049 patients from NCDB and 4278 patients from SEER, all with confirmed osteosarcoma. SEER provides population-based coverage with detailed staging but limited treatment information, while NCDB offers hospital-based data with comprehensive treatment details. We developed separate models for each data set, randomly splitting each into training (80%) and validation (20%) sets. This separation was crucial because it allowed us to test how well our models performed on completely new, unseen data-to test whether a model will work in real-world clinical practice. Primary outcomes included accuracy (proportion of correct predictions), area under the receiver operating characteristic curve (AUC) (discriminative ability between survival outcomes, with values > 0.8 indicating good performance), Brier score (probabilistic prediction accuracy, with values < 0.25 indicating useful models), precision (proportion of positive predictions that were correct), recall (sensitivity for identifying actual outcomes), and F1 score (harmonic mean of precision and recall). The median patient age was 22 years in the NCDB versus 17 years in SEER (p = 0.005), with similar sex distributions (56% male in NCDB, 56% male in SEER) but different racial compositions and overall survival rates (72% and 52% at 2 and 5 years, respectively, for NCDB versus 65% and 43% for SEER). RESULTS: Internal validation showed excellent performance: NCDB-trained models achieved an AUC of 0.93 (95% confidence interval [CI] 0.92 to 0.94) at 2 years and 0.91 (95% CI 0.90 to 0.92) at 5 years, while SEER-trained models achieved 0.90 (95% CI 0.89 to 0.91) and 0.92 (95% CI 0.91 to 0.92), respectively. These AUC values > 0.90 indicate excellent discriminative ability; the models can reliably distinguish between patients who will survive and those who will not. The small differences between NCDB and SEER models (95% CI 0.90 to 0.93) are not clinically meaningful given overlapping confidence intervals. However, external validation revealed poor transferability: NCDB models tested on SEER data achieved an AUC of 0.67 (95% CI 0.65 to 0.68) and 0.60 (95% CI 0.58 to 0.62), while SEER models tested on NCDB data achieved 0.61 (95% CI 0.59 to 0.62) and 0.56 (95% CI 0.55 to 0.58). These external validation AUC values < 0.70 indicate poor predictive performance (barely better than chance and unsuitable for clinical decision-making). This dramatic performance drop demonstrates that models cannot be reliably transferred between different healthcare databases. NCDB models prioritized treatment variables, while SEER models emphasized demographic factors, reflecting the different clinical information captured by each database and explaining why cross-database application fails. CONCLUSION: Models should be validated within the same database environment where they will be applied. These results highlight differences between the NCDB and SEER data sets, showing that models learn database-specific patterns rather than generalizable disease patterns. Cross-database application of models leads to poor predictive performance and should be avoided without revalidation. LEVEL OF EVIDENCE: Level III, prognostic study.

来源:Clin Orthop Relat Res,查看原文。版权归原作者所有。

引用

  1. Girgis AG, Galoaa BM, Goh MH, et al. Bias or Best Fit? A Comparative Analysis of the SEER and NCDB Data Sets in Single-model Machine Learning for Predicting Osteosarcoma Survival Outcomes. Clin Orthop Relat Res. 2025. PMID: 41056069. doi:10.1097/CORR.0000000000003701

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。