乳腺X线摄影中的自动化偏差:人工智能BI-RADS建议对阅片者表现的影响
该前瞻性实验让27名放射科医生在AI系统辅助下阅读50张乳腺X线摄影并给出BI-RADS评估,其中40张中有12张AI建议了错误的BI-RADS分类。结果显示,无论经验多少,阅片者正确率均受AI预测正确性显著影响,且经验不足者更易在AI高估BI-RADS时跟随其建议。
要点
- AI建议的BI-RADS分类正确性显著影响所有经验层次阅片者的正确率。
- 经验不足者更易在AI错误高估BI-RADS时跟随其建议,偏差程度大于中等和丰富经验者。
- 日常阅片需警惕AI建议带来的自动化偏差,尤其对经验不足者。
核心结论
- 经验不足、中等经验和非常丰富的放射科医生在AI系统辅助阅读乳腺X线摄影时均易出现自动化偏差。
- 为确保人机结合的安全部署和准确诊断性能,必须考虑人机交互的这种效应及其他效应。
研究设计与样本
- 前瞻性实验,27名放射科医生阅读50张乳腺X线摄影(2017年1月至2019年12月采集),并提供 purported AI 系统辅助下的BI-RADS评估。
- 图像分两个随机组:第一组为10张训练集,AI建议正确的BI-RADS分类;第二组为40张,其中12张AI建议了错误的BI-RADS分类。
- 评估指标包括阅片者表现、BI-RADS评分偏差程度、对AI系统感知准确性和阅片者对自身BI-RADS评分的信心,采用方差分析、重复测量方差分析及事后检验、Kruskal-Wallis检验及Dunn事后检验。
诊断标准与关键数值
- 经验不足者正确评分乳腺X线摄影百分比:AI正确时平均79.7%±11.7(SD),AI错误时19.8%±14.0;P < .001;r = 0.93。
- 中等经验者正确评分百分比:AI正确时81.3%±10.1,AI错误时24.8%±11.6;P < .001;r = 0.96。
- 非常丰富经验者正确评分百分比:AI正确时82.3%±4.2,AI错误时45.5%±9.1;P = .003;r = 0.97。
- AI错误高估BI-RADS时,经验不足者偏差程度(4.0±1.8)显著高于中等经验者(2.4±1.5;P = .044;r = 0.46)和非常丰富经验者(1.2±0.8;P = .009;r = 0.65)。
临床应用与阅片要点
- 阅片时应意识到AI建议可能引入自动化偏差,尤其在AI建议与自身判断不一致时需独立复核。
- 对经验不足的阅片者,需特别警惕AI高估BI-RADS分类时跟随建议的倾向。
- 在报告中记录AI辅助情况时,应考虑人机交互对最终BI-RADS评分的影响。
局限性
- 研究为实验设置,使用 purported AI 系统,可能不完全代表临床实际AI系统。
- 样本量27名放射科医生,可能限制结果的普遍性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Automation bias (the propensity for humans to favor suggestions from automated decision-making systems) is a known source of error in human-machine interactions, but its implications regarding artificial intelligence (AI)-aided mammography reading are unknown. Purpose To determine how automation bias can affect inexperienced, moderately experienced, and very experienced radiologists when reading mammograms with the aid of an artificial intelligence (AI) system. Materials and Methods In this prospective experiment, 27 radiologists read 50 mammograms and provided their Breast Imaging Reporting and Data System (BI-RADS) assessment assisted by a purported AI system. Mammograms were obtained between January 2017 and December 2019 and were presented in two randomized sets. The first was a training set of 10 mammograms, with the correct BI-RADS category suggested by the AI system. The second was a set of 40 mammograms in which an incorrect BI-RADS category was suggested for 12 mammograms. Reader performance, degree of bias in BI-RADS scoring, perceived accuracy of the AI system, and reader confidence in their own BI-RADS ratings were assessed using analysis of variance (ANOVA) and repeated-measures ANOVA followed by post hoc tests and Kruskal-Wallis tests followed by the Dunn post hoc test. Results The percentage of correctly rated mammograms by inexperienced (mean, 79.7% ± 11.7 [SD] vs 19.8% ± 14.0; P < .001; r = 0.93), moderately experienced (mean, 81.3% ± 10.1 vs 24.8% ± 11.6; P < .001; r = 0.96), and very experienced (mean, 82.3% ± 4.2 vs 45.5% ± 9.1; P = .003; r = 0.97) radiologists was significantly impacted by the correctness of the AI prediction of BI-RADS category. Inexperienced radiologists were significantly more likely to follow the suggestions of the purported AI when it incorrectly suggested a higher BI-RADS category than the actual ground truth compared with both moderately (mean degree of bias, 4.0 ± 1.8 vs 2.4 ± 1.5; P = .044; r = 0.46) and very (mean degree of bias, 4.0 ± 1.8 vs 1.2 ± 0.8; P = .009; r = 0.65) experienced readers. Conclusion The results show that inexperienced, moderately experienced, and very experienced radiologists reading mammograms are prone to automation bias when being supported by an AI-based system. This and other effects of human and machine interaction must be considered to ensure safe deployment and accurate diagnostic performance when combining human readers and AI. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Baltzer in this issue.
引用
- Dratsch, Chen, Rezazade Mehrizi, et al. Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology. 2023. PMID: 37129490. doi:10.1148/radiol.222176
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。