文献速递胸部技术与安全DR

深度学习自动识别具有可转诊异常的胸部X线片:需要重新校准

Automated identification of chest radiographs with referable abnormality with deep learning: need for recalibration.
PubMed 文献 · Eur Radiol · 2020-07-14

本研究纳入1135例急诊连续患者的胸部X线片,评估商用深度学习模型识别异常的校准度,并用8种方法基于概率评分重新校准。原模型倾向于高估异常可能性(平均校准误差0.069,最大校准误差0.179,AUROC 0.949);重新校准后8种方法平均校准误差显著改善(范围0.015-0.036),6种方法AUROC保持(0.909-0.949)。

要点

  • 深度学习模型倾向于高估胸片异常的可能性,平均校准误差0.069。
  • 简单重新校准可显著改善平均校准误差(范围0.015-0.036)。
  • 6种重新校准方法AUROC保持(0.909-0.949)。

核心结论

  • 深度学习算法的校准可通过简单的重新校准程序得到改善。
  • 改善校准可能增强模型对用户的可解释性和可信度。

研究设计与样本

  • 诊断队列研究,纳入1135例连续急诊患者(男:女=582:553;平均年龄52.6岁)的胸部X线片。
  • 使用商用深度学习模型识别异常胸片,每张胸片输出连续概率评分。
  • 基于681张随机抽样胸片对原模型输出进行重新校准,剩余454张胸片用于验证。
  • 在1000次重复随机拆分数据集中评估Brier评分、平均与最大校准误差、绝对Spiegelhalter's Z值及AUROC。

诊断标准与关键数值

  • 原模型平均校准误差0.069,最大校准误差0.179,绝对Spiegelhalter's Z值2.349,AUROC 0.949。
  • 重新校准后8种方法平均校准误差显著改善(范围0.015-0.036),5种方法最大校准误差显著改善(范围0.057-0.172)。
  • 仅1种方法(重新校准常数)绝对Spiegelhalter's Z值显著改善(范围0.809-4.439)。
  • 6种方法AUROC保持(0.909-0.949)。

临床应用与阅片要点

  • 原文结论未给出具体临床操作建议。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVES: To evaluate the calibration of a deep learning (DL) model in a diagnostic cohort and to improve model's calibration through recalibration procedures.

METHODS: Chest radiographs (CRs) from 1135 consecutive patients (M:F = 582:553; mean age, 52.6 years) who visited our emergency department were included. A commercialized DL model was utilized to identify abnormal CRs, with a continuous probability score for each CR. After evaluation of the model calibration, eight different methods were used to recalibrate the original model based on the probability score. The original model outputs were recalibrated using 681 randomly sampled CRs and validated using the remaining 454 CRs. The Brier score for overall performance, average and maximum calibration error, absolute Spiegelhalter's Z for calibration, and area under the receiver operating characteristic curve (AUROC) for discrimination were evaluated in 1000-times repeated, randomly split datasets.

RESULTS: The original model tended to overestimate the likelihood for the presence of abnormalities, exhibiting average and maximum calibration error of 0.069 and 0.179, respectively; an absolute Spiegelhalter's Z value of 2.349; and an AUROC of 0.949. After recalibration, significant improvements in the average (range, 0.015-0.036) and maximum (range, 0.057-0.172) calibration errors were observed in eight and five methods, respectively. Significant improvement in absolute Spiegelhalter's Z (range, 0.809-4.439) was observed in only one method (the recalibration constant). Discriminations were preserved in six methods (AUROC, 0.909-0.949).

CONCLUSION: The calibration of DL algorithm can be augmented through simple recalibration procedures. Improved calibration may enhance the interpretability and credibility of the model for users.

KEY POINTS: • A deep learning model tended to overestimate the likelihood of the presence of abnormalities in chest radiographs. • Simple recalibration of the deep learning model using output scores could improve the calibration of model while maintaining discrimination. • Improved calibration of a deep learning model may enhance the interpretability and the credibility of the model for users.

来源:Eur Radiol,查看原文。版权归原作者所有。

引用

  1. Hwang EJ, Kim H, Lee JH, et al. Automated identification of chest radiographs with referable abnormality with deep learning: need for recalibration. Eur Radiol. 2020. PMID: 32661584. doi:10.1007/s00330-020-07062-7

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。