商业AI软件在低患病率健康筛查环境中分析胸片的多中心外部验证
该研究在两家初级保健中心连续收集3047张胸片,以CT为参考标准,外部验证一款商业AI软件的性能。临床显著病变患病率为2.2%,AI的AUROC为0.648,敏感度35.3%,特异度94.2%;四位放射科住院医师在AI辅助前后阅片,AI仅对经验较少者有小幅提升,但延长了阅片时间。
要点
- 在低患病率筛查人群中,商业AI的敏感度仅35.3%,AUROC 0.648,低于预期。
- AI漏诊的病灶往往比真阳性病灶更小。
- AI辅助仅小幅提升经验较少读者的性能,但延长所有读者阅片时间。
核心结论
- 商业AI软件在高通量、低患病率健康筛查环境中性能低于预期,尽管对经验较少的读者有适度提升。
- 实验环境中AI的技术优势及监管批准可能无法直接转化到真实世界实践。
研究设计与样本
- 多中心外部验证研究,连续收集2018年1月至12月两家初级保健中心的3047张胸片,疾病患病率低。
- 所有胸片根据CT表现标记为正常或异常。
- 四位放射科住院医师在有无AI辅助下分别阅读所有胸片两次,比较AI及读者的AUROC、敏感度和特异度。
关键影像表现/征象
- 临床显著病变患病率为2.2%(68/3047)。
- AI检测到41例肺炎中的12例、5例结核中的3例、22例肿瘤中的9例。
- AI漏诊的病灶倾向于比真阳性病灶更小。
- 读者AUROC无AI时为0.534-0.676,有AI时为0.571-0.688(所有p值<0.05)。
- 所有读者平均阅片时间在AI辅助下延长2.96-10.27秒(所有p值<0.05)。
诊断标准与关键数值
- AI的AUROC为0.648(95%CI 0.630-0.665),敏感度35.3%(95%CI 24.7-47.8),特异度94.2%(95%CI 93.3-95.0)。
临床应用与阅片要点
- 在低患病率筛查环境中使用该商业AI软件时需谨慎评估其敏感度,避免漏诊小病灶。
- 若将AI软件应用于与训练环境不同的特定临床场景,需调整阈值或使用反映该环境的数据进行额外训练。
局限性
- 研究为回顾性设计,仅涉及两家初级保健中心。
- 原文未报告其他局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: To externally validate the performance of a commercial AI software program for interpreting CXRs in a large, consecutive, real-world cohort from primary healthcare centres.
METHODS: A total of 3047 CXRs were collected from two primary healthcare centres, characterised by low disease prevalence, between January and December 2018. All CXRs were labelled as normal or abnormal according to CT findings. Four radiology residents read all CXRs twice with and without AI assistance. The performances of the AI and readers with and without AI assistance were measured in terms of area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity.
RESULTS: The prevalence of clinically significant lesions was 2.2% (68 of 3047). The AUROC, sensitivity, and specificity of the AI were 0.648 (95% confidence interval [CI] 0.630-0.665), 35.3% (CI, 24.7-47.8), and 94.2% (CI, 93.3-95.0), respectively. AI detected 12 of 41 pneumonia, 3 of 5 tuberculosis, and 9 of 22 tumours. AI-undetected lesions tended to be smaller than true-positive lesions. The readers' AUROCs ranged from 0.534-0.676 without AI and 0.571-0.688 with AI (all p values < 0.05). For all readers, the mean reading time was 2.96-10.27 s longer with AI assistance (all p values < 0.05).
CONCLUSIONS: The performance of commercial AI in these high-volume, low-prevalence settings was poorer than expected, although it modestly boosted the performance of less-experienced readers. The technical prowess of AI demonstrated in experimental settings and approved by regulatory bodies may not directly translate to real-world practice, especially where the demand for AI assistance is highest.
KEY POINTS: • This study shows the limited applicability of commercial AI software for detecting abnormalities in CXRs in a health screening population. • When using AI software in a specific clinical setting that differs from the training setting, it is necessary to adjust the threshold or perform additional training with such data that reflects this environment well. • Prospective test accuracy studies, randomised controlled trials, or cohort studies are needed to examine AI software to be implemented in real clinical practice.
引用
- Kim, Yang, Park, et al. Multicentre external validation of a commercial artificial intelligence software to analyse chest radiographs in health screening environments with low disease prevalence. Eur Radiol. 2023. PMID: 36624227. doi:10.1007/s00330-022-09315-z
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。