文献速递母胎医学产科

临床医生与人工智能交互检测胎儿房室间隔缺损:如何优化协作表现

Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance?
PubMed 文献 · Ultrasound Obstet Gynecol · 2024-06-03

研究用121130张胎儿四腔心图像训练AI模型识别房室间隔缺损(AVSD),并招募10名临床医生在单独阅片、二分类AI输出、AI置信度、grad-CAM叠加四种条件下判读2000张图像。二分类AI输出使准确率显著提高至0.865(P<0.001),但错误AI建议使准确率从0.761降至0.693(P<0.001),附加置信度或grad-CAM进一步恶化。

要点

  • 二分类AI输出使临床医生诊断准确率显著提高至0.865(P<0.001)。
  • 错误AI建议使总体准确率从0.761降至0.693(P<0.001)。
  • 附加模型置信度或grad-CAM叠加未改善表现,反而恶化错误AI图像的表现。

核心结论

  • AI与临床医生协作有潜力改善表现,即使模型性能未达专家水平。
  • 提供模型置信度和类激活图叠加等额外信息未改善总体表现,且对AI模型判断错误的图像反而恶化表现。

研究设计与样本

  • 回顾性队列,使用173段超声扫描视频提取的121130张胎儿四腔心图像训练ResNet50模型分类房室间隔缺损(AVSD),其中98例正常心脏、75例AVSD。
  • 从胎儿心脏病中心招募10名临床医生(2名胎儿心脏病顾问、3名儿科心脏病学员、5名胎儿心脏超声技师),每人随机顺序判读2000张胎儿四腔心图像(1000张正常、1000张AVSD)。
  • 数据集含500张图像,每张在四种条件下展示:单独图像无AI输出、图像加二分类AI分类、图像加AI模型置信度、图像加grad-CAM图像叠加。

主要发现

  • AI模型单独准确率为0.798(95%CI 0.760-0.832),敏感度0.868(95%CI 0.834-0.902),特异度0.728(95%CI 0.702-0.754)。
  • 临床医生无AI时准确率为0.844(95%CI 0.834-0.854),敏感度0.827(95%CI 0.795-0.858),特异度0.861(95%CI 0.828-0.895)。
  • 展示二分类AI模型输出使准确率显著提高至0.865(P<0.001),该效应在经验丰富和较不丰富的参与者中均可见。
  • 给出错误AI建议使总体准确率从0.761显著降至0.693(P<0.001),由临床医生I型和II型错误均增加驱动。
  • 展示模型置信度时准确率为0.649(P<0.001),展示grad-CAM时准确率为0.644(P<0.001),均使错误AI建议的恶化效应加剧。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: Artificial intelligence (AI) has shown promise in improving the performance of fetal ultrasound screening in detecting congenital heart disease (CHD). The effect of giving AI advice to human operators has not been studied in this context. Giving additional information about AI model workings, such as confidence scores for AI predictions, may be a way of further improving performance. Our aims were to investigate whether AI advice improved overall diagnostic accuracy (using a single CHD lesion as an exemplar), and to determine what, if any, additional information given to clinicians optimized the overall performance of the clinician-AI team. METHODS: An AI model was trained to classify a single fetal CHD lesion (atrioventricular septal defect (AVSD)), using a retrospective cohort of 121 130 cardiac four-chamber images extracted from 173 ultrasound scan videos (98 with normal hearts, 75 with AVSD); a ResNet50 model architecture was used. Temperature scaling of model prediction probability was performed on a validation set, and gradient-weighted class activation maps (grad-CAMs) produced. Ten clinicians (two consultant fetal cardiologists, three trainees in pediatric cardiology and five fetal cardiac sonographers) were recruited from a center of fetal cardiology to participate. Each participant was shown 2000 fetal four-chamber images in a random order (1000 normal and 1000 AVSD). The dataset comprised 500 images, each shown in four conditions: (1) image alone without AI output; (2) image with binary AI classification; (3) image with AI model confidence; and (4) image with grad-CAM image overlays. The clinicians were asked to classify each image as normal or AVSD. RESULTS: A total of 20 000 image classifications were recorded from 10 clinicians. The AI model alone achieved an accuracy of 0.798 (95% CI, 0.760-0.832), a sensitivity of 0.868 (95% CI, 0.834-0.902) and a specificity of 0.728 (95% CI, 0.702-0.754), and the clinicians without AI achieved an accuracy of 0.844 (95% CI, 0.834-0.854), a sensitivity of 0.827 (95% CI, 0.795-0.858) and a specificity of 0.861 (95% CI, 0.828-0.895). Showing a binary (normal or AVSD) AI model output resulted in significant improvement in accuracy to 0.865 (P < 0.001). This effect was seen in both experienced and less-experienced participants. Giving incorrect AI advice resulted in a significant deterioration in overall accuracy, from 0.761 to 0.693 (P < 0.001), which was driven by an increase in both Type-I and Type-II errors by the clinicians. This effect was worsened by showing model confidence (accuracy, 0.649; P < 0.001) or grad-CAM (accuracy, 0.644; P < 0.001). CONCLUSIONS: AI has the potential to improve performance when used in collaboration with clinicians, even if the model performance does not reach expert level. Giving additional information about model workings such as model confidence and class activation map image overlays did not improve overall performance, and actually worsened performance for images for which the AI model was incorrect. © 2024 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

来源:Ultrasound Obstet Gynecol,查看原文。版权归原作者所有。

引用

  1. Day TG, Matthew J, Budd SF, et al. Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance? Ultrasound Obstet Gynecol. 2024. PMID: 38197584. doi:10.1002/uog.27577

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。