深度学习气胸检测:放射科医师标注与自然语言处理标注训练模型的比较
本研究基于NIH ChestX-ray14数据集,比较了用放射科医师标注和NLP标注训练的三个CNN模型(ResNet-50、DenseNet-121、EfficientNetB3)检测气胸的性能。结果显示,医师标注模型在NIH内部测试集和外部测试集上的AUC均显著高于NLP标注模型,表明医师标注能提升模型性能和泛化能力。
要点
- 医师标注训练的模型在内部和外部测试集上AUC均更高。
- EfficientNetB3在医师标注下外部测试集AUC达0.915。
- 标注质量影响深度学习气胸检测的泛化能力。
核心结论
- 用放射科医师标注训练的气胸深度学习模型比用NLP标注训练的模型性能更好、泛化能力更强。
研究设计与样本
- 回顾性研究使用NIH ChestX-ray14数据集,含112,120张正面胸部X线片,分别用NLP标注(数据集A)和4名放射科医师视觉确认标注(数据集B)训练三种CNN架构。
- 模型在官方NIH测试集和本院急诊科525张胸部X线片的外部测试集上评估AUC。
诊断标准与关键数值
- NIH内部测试集AUC:NLP标注/医师标注模型分别为ResNet-50 0.838(95%CI:0.830,0.846)/0.881(95%CI:0.873,0.887)(p=0.034),DenseNet-121 0.839(95%CI:0.831,0.847)/0.880(95%CI:0.873,0.887),EfficientNetB3 0.869(95%CI:0.863,0.876)/0.943(95%CI:0.939,0.946)(p≤0.001)。
- 外部测试集AUC:NLP标注/医师标注模型分别为ResNet-50 0.686(95%CI:0.632,0.740)/0.806(95%CI:0.758,0.854),DenseNet-121 0.736(95%CI:0.686,0.787)/0.871(95%CI:0.830,0.912),EfficientNetB3 0.822(95%CI:0.775,0.868)/0.915(95%CI:0.882,0.948),所有架构p<0.001。
临床应用与阅片要点
- 在日常阅片中,使用医师标注训练的气胸检测模型可能提供更可靠的辅助诊断。
- 注意模型性能在外部数据上可能下降,需结合临床评估。
局限性
- 外部测试集仅来自单中心急诊科,样本量有限。
- 研究为回顾性设计,未报告前瞻性验证。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: To compare the performance of pneumothorax deep learning detection models trained with radiologist versus natural language processing (NLP) labels on the NIH ChestX-ray14 dataset.
MATERIALS AND METHODS: The ChestX-ray14 dataset consisted of 112,120 frontal chest radiographs with 5302 positive and 106, 818 negative labels for pneumothorax using NLP (dataset A). All 112,120 radiographs were also inspected by 4 radiologists leaving a visually confirmed set of 5,138 positive and 104,751 negative for pneumothorax (dataset B). Datasets A and B were used independently to train 3 convolutional neural network (CNN) architectures (ResNet-50, DenseNet-121 and EfficientNetB3). All models' area under the receiver operating characteristic curve (AUC) were evaluated with the official NIH test set and an external test set of 525 chest radiographs from our emergency department.
RESULTS: There were significantly higher AUCs on the NIH internal test set for CNN models trained with radiologist vs NLP labels across all architectures. AUCs for the NLP/radiologist-label models were 0.838 (95%CI:0.830, 0.846)/0.881 (95%CI:0.873,0.887) for ResNet-50 (p = 0.034), 0.839 (95%CI:0.831,0.847)/0.880 (95%CI:0.873,0.887) for DenseNet-121, and 0.869 (95%CI: 0.863,0.876)/0.943 (95%CI: 0.939,0.946) for EfficientNetB3 (p ≤0.001). Evaluation with the external test set also showed higher AUCs (p <0.001) for the CNN models trained with radiologist versus NLP labels across all architectures. The AUCs for the NLP/radiologist-label models were 0.686 (95%CI:0.632,0.740)/0.806 (95%CI:0.758,0.854) for ResNet-50, 0.736 (95%CI:0.686, 0.787)/0.871 (95%CI:0.830,0.912) for DenseNet-121, and 0.822 (95%CI: 0.775,0.868)/0.915 (95%CI: 0.882,0.948) for EfficientNetB3.
CONCLUSION: We demonstrated improved performance and generalizability of pneumothorax detection deep learning models trained with radiologist labels compared to models trained with NLP labels.
引用
- Hallinan JTPD, Feng M, Ng D, et al. Detection of Pneumothorax with Deep Learning Models: Learning From Radiologist Labels vs Natural Language Processing Model Generated Labels. Acad Radiol. 2021. PMID: 34649780. doi:10.1016/j.acra.2021.09.013
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。