文献速递胸部技术与安全DR

医学图像分类中的类别不平衡:面向判别与校准性能的更优评估实践

Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance.
PubMed 文献 · Eur Radiol · 2024-06-11

该研究以胸部X线为案例,通过文献综述和两个大型数据集实验,评估医学影像分类器在类别不平衡下的常用评估实践。文献确认社区常使用被多数类主导的指标且校准研究少见;实验表明当前评估实践可能无法反映真实临床性能,建议补充AUC-PR、调整AUC-PR和平衡Brier分数等指标。

要点

  • 现有X线CAD文献常用被多数类主导的指标,可能掩盖少数类性能不足。
  • 校准研究在胸部X线分类器中仍不常见,但医疗场景中很重要。
  • 建议补充AUC-PR、调整AUC-PR和平衡Brier分数以更准确反映临床性能。

核心结论

  • 当前评估实践在类别不平衡的胸部X线分类器中可能无法反映真实临床场景下的模型性能。
  • 建议纳入AUC-PR、调整AUC-PR和平衡Brier分数等补充指标,同时反映判别与校准性能。

研究设计与样本

  • 研究类型包括简要文献综述和系统实验,以胸部X线为案例。
  • 系统实验在两个主要胸部X线数据集上进行,考察不同类别比例下多种性能指标的行为。
  • 文献综述检查了评估X线分类器时 prevailing 的科学实践,未报告样本量、中心数或设备/序列细节。

诊断标准与关键数值

  • 文献综述确认社区倾向于使用被多数类主导的指标,即使处理高度不平衡数据集。
  • 文献综述确认胸部X线分类器中包含校准研究仍不常见。
  • 系统实验确认当前评估实践可能不反映真实临床场景中的模型性能。
  • 推荐补充指标包括AUC-PR、调整AUC-PR和平衡Brier分数,但原文未报告具体数值、阈值或分级标准。

临床应用与阅片要点

  • 日常阅片和报告相关模型评估时,应避免仅依赖被多数类主导的指标,需关注少数类性能。
  • 在类别不平衡场景下,建议补充AUC-PR、调整AUC-PR和平衡Brier分数以更全面评估模型。
  • 注意校准性能的重要性,避免仅关注判别能力而忽略模型校准。

局限性

  • 分析以胸部X线为案例,结论可能不直接推广到其他影像模态或任务。
  • 文献综述为简要回顾,未报告系统检索范围或纳入文献数量。
  • 系统实验仅涉及两个胸部X线数据集,未报告多中心外部验证。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: This work aims to assess standard evaluation practices used by the research community for evaluating medical imaging classifiers, with a specific focus on the implications of class imbalance. The analysis is performed on chest X-rays as a case study and encompasses a comprehensive model performance definition, considering both discriminative capabilities and model calibration.

MATERIALS AND METHODS: We conduct a concise literature review to examine prevailing scientific practices used when evaluating X-ray classifiers. Then, we perform a systematic experiment on two major chest X-ray datasets to showcase a didactic example of the behavior of several performance metrics under different class ratios and highlight how widely adopted metrics can conceal performance in the minority class.

RESULTS: Our literature study confirms that: (1) even when dealing with highly imbalanced datasets, the community tends to use metrics that are dominated by the majority class; and (2) it is still uncommon to include calibration studies for chest X-ray classifiers, albeit its importance in the context of healthcare. Moreover, our systematic experiments confirm that current evaluation practices may not reflect model performance in real clinical scenarios and suggest complementary metrics to better reflect the performance of the system in such scenarios.

CONCLUSION: Our analysis underscores the need for enhanced evaluation practices, particularly in the context of class-imbalanced chest X-ray classifiers. We recommend the inclusion of complementary metrics such as the area under the precision-recall curve (AUC-PR), adjusted AUC-PR, and balanced Brier score, to offer a more accurate depiction of system performance in real clinical scenarios, considering metrics that reflect both, discrimination and calibration performance.

CLINICAL RELEVANCE STATEMENT: This study underscores the critical need for refined evaluation metrics in medical imaging classifiers, emphasizing that prevalent metrics may mask poor performance in minority classes, potentially impacting clinical diagnoses and healthcare outcomes.

KEY POINTS: Common scientific practices in papers dealing with X-ray computer-assisted diagnosis (CAD) systems may be misleading. We highlight limitations in reporting of evaluation metrics for X-ray CAD systems in highly imbalanced scenarios. We propose adopting alternative metrics based on experimental evaluation on large-scale datasets.

来源:Eur Radiol,查看原文。版权归原作者所有。

引用

  1. Mosquera C, Ferrer L, Milone DH, et al. Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance. Eur Radiol. 2024. PMID: 38861161. doi:10.1007/s00330-024-10834-0

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。