从视觉变换器集成中蒸馏知识以改进乳腺超声分类
本研究基于公开乳腺超声图像(BUSI)数据集,比较视觉变换器(ViT)与卷积神经网络(CNN)对乳腺超声图像良恶性分类的性能,并评估自监督与监督初始化、10个独立训练ViT的集成以及知识蒸馏得到的单一ViT。集成模型测试集AuROC为0.977、AuPRC为0.965,优于独立训练ViT的平均值(AuROC 0.958 ± 0.05,AuPRC 0.931 ± 0.016);蒸馏ViT的AuROC为0.972、AuPRC为0.960。
要点
- 集成10个ViT在测试集上AuROC 0.977、AuPRC 0.965,优于独立ViT平均AuROC 0.958 ± 0.05、AuPRC 0.931 ± 0.016。
- 蒸馏得到的单一ViT测试集AuROC 0.972、AuPRC 0.960。
- 自监督ViT性能优于监督初始化和随机初始化ViT,ViT优于CNN。
核心结论
- 迁移学习和集成学习各自可独立提升性能,且可序贯结合共同提升最终模型性能;单一视觉变换器可通过知识蒸馏训练以匹配一组视觉变换器集成的性能。
研究设计与样本
- 基于公开乳腺超声图像(BUSI)数据集构建单视图B模式超声图像深度学习分类模型,采用五折交叉验证训练;比较ViT与CNN、监督与自监督及随机初始化ViT、10个独立训练ViT的集成(输出未加权平均)与单个ViT,并用知识蒸馏训练单一ViT以模拟集成ViT。
诊断标准与关键数值
- 测试集上集成模型AuROC 0.977、AuPRC 0.965;独立训练ViT平均AuROC 0.958 ± 0.05、平均AuPRC 0.931 ± 0.016;蒸馏ViT的AuROC 0.972、AuPRC 0.960。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: To develop a deep learning model for the automated classification of breast ultrasound images as benign or malignant. More specifically, the application of vision transformers, ensemble learning, and knowledge distillation is explored for breast ultrasound classification.
MATERIALS AND METHODS: Single view, B-mode ultrasound images were curated from the publicly available Breast Ultrasound Image (BUSI) dataset, which has categorical ground truth labels (benign vs malignant) assigned by radiologists and malignant cases confirmed by biopsy. The performance of vision transformers (ViT) is compared to convolutional neural networks (CNN), followed by a comparison between supervised, self-supervised, and randomly initialized ViT. Subsequently, the ensemble of 10 independently trained ViT, where the ensemble model is the unweighted average of the output of each individual model is compared to the performance of each ViT alone. Finally, we train a single ViT to emulate the ensembled ViT using knowledge distillation.
RESULTS: On this dataset that was trained using five-fold cross validation, ViT outperforms CNN, while self-supervised ViT outperform supervised and randomly initialized ViT. The ensemble model achieves an area under the receiver operating characteristics curve (AuROC) and area under the precision recall curve (AuPRC) of 0.977 and 0.965 on the test set, outperforming the average AuROC and AuPRC of the independently trained ViTs (0.958 ± 0.05 and 0.931 ± 0.016). The distilled ViT achieves an AuROC and AuPRC of 0.972 and 0.960.
CONCLUSION: Both transfer learning and ensemble learning can each offer increased performance independently and can be sequentially combined to collectively improve the performance of the final model. Furthermore, a single vision transformer can be trained to match the performance of an ensemble of a set of vision transformers using knowledge distillation.
引用
- Zhou G, Mosadegh B. Distilling Knowledge From an Ensemble of Vision Transformers for Improved Classification of Breast Ultrasound. Acad Radiol. 2023. PMID: 37666747. doi:10.1016/j.acra.2023.08.006
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。