深度学习算法检测颅内出血的诊断准确性与失败模式分析
本研究回顾性纳入本单位3605例急诊成人非增强头颅CT,由神经放射学专科医师与AI决策支持系统Aidoc分别评估颅内出血。AI与医师判读一致率96.9%,总体敏感度92.3%、特异度97.7%、阳性预测值81.3%、阴性预测值99.2%。既往神经外科手术、出血类型及出血数量与模型性能下降显著相关,但失败模式来源未能明确。
要点
- AI与神经放射学医师判读一致率为96.9%。
- 阳性预测值81.3%,低于既往研究报告,需警惕假阳性。
- 既往神经外科手术、出血类型及多发出血与模型性能下降相关。
核心结论
- 在本机构中,Aidoc AI决策支持系统诊断颅内出血的准确率低于既往报告,阳性预测值意外偏低。
- 研究未能明确导致性能差异的失败模式来源,提示该类工具的可推广性存在不确定性。
- 需要标准化研究设计以实现严格且可重复的多中心深度学习技术比较。
研究设计与样本
- 女性占51%,平均年龄61±21岁;每例扫描由具备附加资格认证的神经放射学专科医师与Aidoc分别评估颅内出血。
- 参考标准为神经放射学专科医师判读,并采用定量CT影像组学进行失败模式分析。
诊断标准与关键数值
- Aidoc与神经放射学医师判读一致率为96.9%。
- 总体敏感度92.3%、特异度97.7%、阳性预测值81.3%、阴性预测值99.2%。
临床应用与阅片要点
- AI判读阳性时需结合既往神经外科手术史、出血类型及出血数量综合判断,警惕假阳性。
- AI阴性预测值较高,但阳性预测值偏低,报告时应关注AI标记与医师判读不一致的病例。
局限性
- 单中心、回顾性研究设计,结果外推需谨慎。
- 定量CT影像组学未能揭示一致与不一致研究之间的显著差异,失败模式来源未明。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVE: To determine the institutional diagnostic accuracy of an artificial intelligence (AI) decision support systems (DSS), Aidoc, in diagnosing intracranial hemorrhage (ICH) on noncontrast head CTs and to assess the potential generalizability of an AI DSS.
METHODS: This retrospective study included 3,605 consecutive, emergent, adult noncontrast head CT scans performed between July 1, 2019, and December 30, 2019, at our institution (51% female subjects, mean age of 61 ± 21 years). Each scan was evaluated for ICH by both a certificate of added qualification certified neuroradiologist and Aidoc. We determined the diagnostic accuracy of the AI model and performed a failure mode analysis with quantitative CT radiomic image characterization.
RESULTS: Of the 3,605 scans, 349 cases of ICH (9.7% of studies) were identified. The neuroradiologist and Aidoc interpretations were concordant in 96.9% of cases and the overall sensitivity, specificity, positive predictive value, and negative predictive value were 92.3%, 97.7%, 81.3%, and 99.2%, respectively, with positive predictive values unexpectedly lower than in previously reported studies. Prior neurosurgery, type of ICH, and number of ICHs were significantly associated with decreased model performance. Quantitative image characterization with CT radiomics failed to reveal significant differences between concordant and discordant studies.
DISCUSSION: This study revealed decreased diagnostic accuracy of an AI DSS at our institution. Despite extensive evaluation, we were unable to identify the source of this discrepancy, raising concerns about the generalizability of these tools with indeterminate failure modes. These results further highlight the need for standardized study design to allow for rigorous and reproducible site-to-site comparison of emerging deep learning technologies.
引用
- Voter AF, Meram E, Garrett JW, et al. Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Intracranial Hemorrhage. J Am Coll Radiol. 2021. PMID: 33819478. doi:10.1016/j.jacr.2021.03.005
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。