深度学习陷阱:新型超声设备引入对算法性能的影响及域适应的现实
本研究前瞻性采集21段下腔静脉超声视频,用两种新型超声设备挑战既往验证过的深度学习算法,并与2名盲法POCUS专家对比。算法在高质量推车式设备上kappa为0.31和0.39(一致性尚可),在低质量智能手机设备上降至-0.09和0.09(低于随机预期);2名专家间kappa为0.88。
要点
- 同一深度学习算法换用新型超声设备后诊断下腔静脉塌陷的kappa从0.66–0.78降至0.31–0.39。
- 低图像质量手持设备上算法kappa为-0.09和0.09,低于随机预期的一致性。
- 2名POCUS专家间kappa为0.88,接近完美一致。
核心结论
- 该既往验证过的深度学习算法在两种新型超声设备图像上性能变差,低图像质量手持设备上的表现远差于高质量推车式设备。
研究设计与样本
- 前瞻性研究,每种新型超声设备各获取21段下腔静脉视频,以2名盲法POCUS专家判定为参考,计算Cohen's kappa评估算法与专家一致性。
诊断标准与关键数值
- 算法与专家在既往同款POCUS设备新患者数据上kappa为0.78(95%CI 0.49-1.0)和0.66(95%CI 0.31-1.0)。
- 高质量推车式POCUS设备上算法kappa为0.31(95%CI 0.19-0.81)和0.39(95%CI 0.11-0.89)。
- 低质量智能手机设备上算法kappa为-0.09(95%CI -0.95至0.76)和0.09(95%CI -0.65至0.82)。
- 2名POCUS专家间关于下腔静脉塌陷的kappa为0.88(95%CI 0.64-1.0)。
临床应用与阅片要点
- 作者指出,既往验证的深度学习算法在引入新型超声设备后性能下降,图像质量较低的手持设备上表现更差。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: To test deep learning (DL) algorithm performance repercussions by introducing novel ultrasound equipment into a clinical setting.
METHODS: Researchers introduced prospectively obtained inferior vena cava (IVC) videos from a similar patient population using novel ultrasound equipment to challenge a previously validated DL algorithm (trained on a common point of care ultrasound [POCUS] machine) to assess IVC collapse. Twenty-one new videos were obtained for each novel ultrasound machine. The videos were analyzed for complete collapse by the algorithm and by 2 blinded POCUS experts. Cohen's kappa was calculated for agreement between the 2 POCUS experts and DL algorithm. Previous testing showed substantial agreement between algorithm and experts with Cohen's kappa of 0.78 (95% CI 0.49-1.0) and 0.66 (95% CI 0.31-1.0) on new patient data using, the same ultrasound equipment.
RESULTS: Challenged with higher image quality (IQ) POCUS cart ultrasound videos, algorithm performance declined with kappa values of 0.31 (95% CI 0.19-0.81) and 0.39 (95% CI 0.11-0.89), showing fair agreement. Algorithm performance plummeted on a lower IQ, smartphone device with a kappa value of -0.09 (95% CI -0.95 to 0.76) and 0.09 (95% CI -0.65 to 0.82), respectively, showing less agreement than would be expected by chance. Two POCUS experts had near perfect agreement with a kappa value of 0.88 (95% CI 0.64-1.0) regarding IVC collapse.
CONCLUSIONS: Performance of this previously validated DL algorithm worsened when faced with ultrasound studies from 2 novel ultrasound machines. Performance was much worse on images from a lower IQ hand-held device than from a superior cart-based device.
引用
- Blaivas M, Blaivas LN, Tsung JW, et al. Deep Learning Pitfall: Impact of Novel Ultrasound Equipment Introduction on Algorithm Performance and the Realities of Domain Adaptation. J Ultrasound Med. 2021. PMID: 34133034. doi:10.1002/jum.15765
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。