一种深度学习算法检测颈椎骨折的诊断准确性与失败模式分析
本回顾性研究纳入1904例成人急诊非增强颈椎CT,评估人工智能决策支持系统Aidoc检测颈椎骨折的性能并进行失败模式分析。Aidoc与神经放射科医师判读一致率为91.5%,正确识别122处骨折中的67处(54.9%),敏感度54.9%(95%CI 45.7%-63.9%),特异度94.1%(95%CI 92.9%-95.1%),阳性预测值38.7%(95%CI 33.1%-44.7%),阴性预测值96.8%(95%CI 96.2%-97.4%)。作者结论认为该人工智能决策支持系统诊断准确性差,许多类似算法缺乏外部验证,对其泛化性、效用和快速部署提出担忧,需在广泛实施前进行严格评估。
要点
- Aidoc检测颈椎骨折敏感度54.9%(95%CI 45.7%-63.9%),特异度94.1%(95%CI 92.9%-95.1%),阳性预测值38.7%(95%CI 33.1%-44.7%)。
- Aidoc与神经放射科医师判读一致率为91.5%,122处骨折中正确识别67处(54.9%),假阳性标记106例。
- 慢性骨折检测性能更差,诊断性能差异未因检查指征或患者特征而改变。
核心结论
- 作者观察到该人工智能决策支持系统检测颈椎骨折的诊断准确性差,并对其泛化性、效用和快速部署提出担忧,认为广泛实施前需进一步严格评估。
研究设计与样本
- 回顾性研究,纳入1904例成人急诊非增强颈椎CT扫描(平均年龄60岁,SD 22岁,男性50.3%)。
- 以Aidoc与神经放射科医师判读确定颈椎骨折存在与否,分歧由独立仲裁解决,计算诊断准确性并进行失败模式分析。
诊断标准与关键数值
- Aidoc检测颈椎骨折敏感度54.9%(95%CI 45.7%-63.9%),特异度94.1%(95%CI 92.9%-95.1%)。
- 阳性预测值38.7%(95%CI 33.1%-44.7%),阴性预测值96.8%(95%CI 96.2%-97.4%)。
- Aidoc与神经放射科医师判读一致率91.5%,122处骨折中正确识别67处(54.9%),假阳性标记106例。
临床应用与阅片要点
- 作者结论认为该人工智能决策支持系统诊断准确性差,许多类似算法缺乏外部验证,对其泛化性、效用和快速部署提出担忧,需在广泛实施前进行进一步严格评估。
局限性
- 作者结论指出许多类似算法也仅获得很少或没有外部验证,本研究对其泛化性和效用提出担忧。
- 作者结论认为需要进一步严格评估以了解这些工具的弱点,之后才能广泛实施。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
BACKGROUND AND PURPOSE: Artificial intelligence decision support systems are a rapidly growing class of tools to help manage ever-increasing imaging volumes. The aim of this study was to evaluate the performance of an artificial intelligence decision support system, Aidoc, for the detection of cervical spinal fractures on noncontrast cervical spine CT scans and to conduct a failure mode analysis to identify areas of poor performance.
MATERIALS AND METHODS: This retrospective study included 1904 emergent noncontrast cervical spine CT scans of adult patients (60 [SD, 22] years, 50.3% men). The presence of cervical spinal fracture was determined by Aidoc and an attending neuroradiologist; discrepancies were independently adjudicated. Algorithm performance was assessed by calculation of the diagnostic accuracy, and a failure mode analysis was performed.
RESULTS: Aidoc and the neuroradiologist's interpretation were concordant in 91.5% of cases. Aidoc correctly identified 67 of 122 fractures (54.9%) with 106 false-positive flagged studies. Diagnostic performance was calculated as the following: sensitivity, 54.9% (95% CI, 45.7%-63.9%); specificity, 94.1% (95% CI, 92.9%-95.1%); positive predictive value, 38.7% (95% CI, 33.1%-44.7%); and negative predictive value, 96.8% (95% CI, 96.2%-97.4%). Worsened performance was observed in the detection of chronic fractures; differences in diagnostic performance were not altered by study indication or patient characteristics.
CONCLUSIONS: We observed poor diagnostic accuracy of an artificial intelligence decision support system for the detection of cervical spine fractures. Many similar algorithms have also received little or no external validation, and this study raises concerns about their generalizability, utility, and rapid pace of deployment. Further rigorous evaluations are needed to understand the weaknesses of these tools before widespread implementation.
引用
- Voter AF, Larson ME, Garrett JW, et al. Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Cervical Spine Fractures. AJNR Am J Neuroradiol. 2021. PMID: 34117018. doi:10.3174/ajnr.A7179
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。