文献速递妇产影像儿科影像心血管US

临床医生与人工智能交互检测胎儿房室间隔缺损的超声表现:如何优化协作性能?

Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance?
PubMed 文献 · Ultrasound Obstet Gynecol · 2024-06-03

本研究使用回顾性队列的121130张胎儿四腔心图像训练ResNet50模型分类房室间隔缺损,并招募10名临床医生在四种条件下(仅图像、图像加AI二分类、加模型置信度、加grad-CAM覆盖图)各判读2000张图像。结果显示仅提供AI二分类输出可显著提高准确率至0.865(P<0.001),但AI判断错误时准确率从0.761降至0.693(P<0.001),显示置信度或grad-CAM后进一步降至0.649和0.644(P<0.001)。

要点

  • 仅提供AI二分类输出使临床医生诊断准确率从0.844提高至0.865(P<0.001),经验丰富与经验较少者均受益。
  • AI判断错误时,临床医生准确率从0.761降至0.693(P<0.001),I型和II型错误均增加。
  • 在AI错误图像上附加模型置信度或grad-CAM覆盖图使准确率进一步降至0.649和0.644(P<0.001)。

核心结论

  • AI与临床医生协作有潜力提升胎儿房室间隔缺损的超声检测性能,即使模型未达专家水平;但提供模型置信度或类激活图覆盖图等额外信息未改善整体性能,且在AI判断错误的图像上反而使性能变差。

研究设计与样本

  • 回顾性队列研究,使用173段超声扫查视频提取的121130张胎儿四腔心图像训练ResNet50模型,其中正常心脏98例、房室间隔缺损75例;对模型预测概率进行温度缩放校准并生成grad-CAM;从胎儿心脏病中心招募10名临床医生(2名胎儿心脏病顾问、3名儿科心脏病受训者、5名胎儿心脏超声医师),每人在四种条件下判读2000张图像(1000正常、1000房室间隔缺损),共记录20000次图像分类。

诊断标准与关键数值

  • AI模型单独诊断准确度0.798(95%CI 0.760-0.832),敏感度0.868(95%CI 0.834-0.902),特异度0.728(95%CI 0.702-0.754)。
  • 临床医生无AI辅助时诊断准确度0.844(95%CI 0.834-0.854),敏感度0.827(95%CI 0.795-0.858),特异度0.861(95%CI 0.828-0.895)。
  • 提供AI二分类输出后准确度提高至0.865(P<0.001)。
  • AI判断错误时,整体准确度从0.761降至0.693(P<0.001);附加模型置信度时准确度0.649(P<0.001),附加grad-CAM时准确度0.644(P<0.001)。

临床应用与阅片要点

  • 作者结论认为AI与临床医生协作有潜力提升性能,但提供模型置信度或类激活图覆盖图等额外信息未改善整体性能且在AI错误时使性能变差。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVES: Artificial intelligence (AI) has shown promise in improving the performance of fetal ultrasound screening in detecting congenital heart disease (CHD). The effect of giving AI advice to human operators has not been studied in this context. Giving additional information about AI model workings, such as confidence scores for AI predictions, may be a way of further improving performance. Our aims were to investigate whether AI advice improved overall diagnostic accuracy (using a single CHD lesion as an exemplar), and to determine what, if any, additional information given to clinicians optimized the overall performance of the clinician-AI team.

METHODS: An AI model was trained to classify a single fetal CHD lesion (atrioventricular septal defect (AVSD)), using a retrospective cohort of 121 130 cardiac four-chamber images extracted from 173 ultrasound scan videos (98 with normal hearts, 75 with AVSD); a ResNet50 model architecture was used. Temperature scaling of model prediction probability was performed on a validation set, and gradient-weighted class activation maps (grad-CAMs) produced. Ten clinicians (two consultant fetal cardiologists, three trainees in pediatric cardiology and five fetal cardiac sonographers) were recruited from a center of fetal cardiology to participate. Each participant was shown 2000 fetal four-chamber images in a random order (1000 normal and 1000 AVSD). The dataset comprised 500 images, each shown in four conditions: (1) image alone without AI output; (2) image with binary AI classification; (3) image with AI model confidence; and (4) image with grad-CAM image overlays. The clinicians were asked to classify each image as normal or AVSD.

RESULTS: A total of 20 000 image classifications were recorded from 10 clinicians. The AI model alone achieved an accuracy of 0.798 (95% CI, 0.760-0.832), a sensitivity of 0.868 (95% CI, 0.834-0.902) and a specificity of 0.728 (95% CI, 0.702-0.754), and the clinicians without AI achieved an accuracy of 0.844 (95% CI, 0.834-0.854), a sensitivity of 0.827 (95% CI, 0.795-0.858) and a specificity of 0.861 (95% CI, 0.828-0.895). Showing a binary (normal or AVSD) AI model output resulted in significant improvement in accuracy to 0.865 (P < 0.001). This effect was seen in both experienced and less-experienced participants. Giving incorrect AI advice resulted in a significant deterioration in overall accuracy, from 0.761 to 0.693 (P < 0.001), which was driven by an increase in both Type-I and Type-II errors by the clinicians. This effect was worsened by showing model confidence (accuracy, 0.649; P < 0.001) or grad-CAM (accuracy, 0.644; P < 0.001).

CONCLUSIONS: AI has the potential to improve performance when used in collaboration with clinicians, even if the model performance does not reach expert level. Giving additional information about model workings such as model confidence and class activation map image overlays did not improve overall performance, and actually worsened performance for images for which the AI model was incorrect. © 2024 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

来源:Ultrasound Obstet Gynecol,查看原文。版权归原作者所有。

引用

  1. Day TG, Matthew J, Budd SF, et al. Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance? Ultrasound Obstet Gynecol. 2024. PMID: 38197584. doi:10.1002/uog.27577

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。