文献速递神经外周血管技术与安全CT

在临床环境中实施人工智能算法:准确性悖论的案例研究

Implementing an AI algorithm in the clinical setting: a case study for the accuracy paradox.
PubMed 文献 · Eur Radiol · 2024-12-31

该研究在单一三级医院急诊科部署大血管闭塞检测算法,对48例CT血管造影患者进行回顾性分析,算法敏感度100%、特异度92%,但假发现率67%、阳性预测值33%,远低于制造商报告的95-98%。作者认为算法被放射科医生认为不准确并撤出临床,可能源于准确性悖论这一认知偏差。

要点

  • 算法敏感度与特异度虽高,但本地阳性预测值仅33%,假发现率67%,远低于制造商报告的阳性预测值95-98%。
  • 本地疾病患病率4.1%远低于制造商报告值45.0-62.2%,是阳性预测值差异的主要原因。
  • 作者认为算法被感知为不准确并撤出临床,可能源于准确性悖论这一认知偏差。

核心结论

  • 尽管大血管闭塞算法性能达到制造商宣传水平,但因假阳性多于预期而被认为不准确并撤出临床,作者认为这可能是准确性悖论这一认知偏差所致。

研究设计与样本

  • 在单一三级医院急诊科部署大血管闭塞检测算法,对2021年9月1日至27日期间48例急诊CT血管造影检查进行回顾性准确性分析。

诊断标准与关键数值

  • 算法检测大血管闭塞的敏感度100%、特异度92%、假发现率67%、阳性预测值33%,制造商报告阳性预测值为95-98%;本地疾病患病率4.1%,制造商报告值为45.0-62.2%。

临床应用与阅片要点

  • 作者认为,在评估和使用人工智能工具时,应向放射科医生提供基于与其实践相似的疾病患病率的指标,以确保对真实世界性能有现实预期。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVES: We report our experience implementing an algorithm for the detection of large vessel occlusion (LVO) for suspected stroke in the emergency setting, including its performance, and offer an explanation as to why it was poorly received by radiologists.

MATERIALS AND METHODS: An algorithm was deployed in the emergency room at a single tertiary care hospital for the detection of LVO on CT angiography (CTA) between September 1st-27th, 2021. A retrospective analysis of the algorithm's accuracy was performed.

RESULTS: During the study period, 48 patients underwent CTA examination in the emergency department to evaluate for emergent LVO, with 2 positive cases (60.3 years ± 18.2; 32 women). The LVO algorithm demonstrated a sensitivity and specificity of 100% and 92%, respectively. While the sensitivity of the algorithm at our institution was even higher than the manufacturer's reported values, the false discovery rate was 67%, leading to the perception that the algorithm was inaccurate. In addition, the positive predictive value at our institution was 33% compared with the manufacturer's reported values of 95-98%. This disparity can be attributed to differences in disease prevalence of 4.1% at our institution compared with 45.0-62.2% from the manufacturer's reported values.

CONCLUSION: Despite the LVO algorithm's accuracy performing as advertised, it was perceived as inaccurate due to more false positives than anticipated and was removed from clinical practice. This was likely due to a cognitive bias called the accuracy paradox. To mitigate the accuracy paradox, radiologists should be presented with metrics based on a disease prevalence similar to their practice when evaluating and utilizing artificial intelligence tools.

KEY POINTS: Question An artificial intelligence algorithm for detecting emergent LVOs was implemented in an emergency department, but it was perceived to be inaccurate. Findings Although the algorithm's accuracy was both high and as advertised, the algorithm demonstrated a high false discovery rate. Clinical relevance The misperception of the algorithm's inaccuracy was likely due to a special case of the base rate fallacy-the accuracy paradox. Equipping radiologists with an algorithm's false discovery rate based on local prevalence will ensure realistic expectations for real-world performance.

来源:Eur Radiol,查看原文。版权归原作者所有。

引用

  1. Scaringi JA, McTaggart RA, Alvin MD, et al. Implementing an AI algorithm in the clinical setting: a case study for the accuracy paradox. Eur Radiol. 2024. PMID: 39741216. doi:10.1007/s00330-024-11332-z

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。