预测变异性用于识别MRI和CT癌症诊断中AI性能下降
该多中心回顾性研究分析了胰腺癌CT检测、前列腺癌MRI检测及肺结节恶性预测三个任务,将AI预测变异性作为不确定性量化指标;在80%确定预测阈值下,确定组准确率较不确定组高21%-29%,肺结节任务中确定组AI与临床医师准确率无显著差异,不确定组AI低于医师。
要点
- AI预测变异性可作为不确定性量化指标,识别诊断性能下降的病例。
- 确定组AI准确率较不确定组高21%-29%,病灶水平pAUC高0.25-0.39。
- 肺结节恶性预测中,不确定组AI准确率50%低于临床医师68%。
核心结论
- 基于集成预测变异性的不确定性量化指标能一致地识别癌症诊断中AI性能下降的病例。
研究设计与样本
- 回顾性分析三个既往研究的多中心数据集和公开AI算法,任务包括增强CT胰腺癌检测、MRI前列腺癌检测和低剂量CT肺结节恶性预测。
- 共使用18022张图像进行训练,838张图像进行测试;每个任务的算法扩展为基于集成预测变异性的不确定性评分。
诊断标准与关键数值
- 在80%确定预测阈值下,确定组准确率较不确定组高21%-29%,较整体测试数据集高4%-6%。
- 确定组病灶水平pAUC较不确定组高0.25-0.39,较整体测试数据集高0.05-0.08(P < .001)。
- 肺结节恶性预测中,确定组AI与临床医师准确率分别为80%(95% CI: 76, 85)与78%(95% CI: 70, 87),差异无统计学意义(P = .07)。
- 肺结节恶性预测中,不确定组AI与临床医师准确率分别为50%(95% CI: 37, 64)与68%(95% CI: 60, 76),AI低于医师(P < .001)。
临床应用与阅片要点
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background A priori identification of patients at risk of artificial intelligence (AI) failure in diagnosing cancer would contribute to the safer clinical integration of diagnostic algorithms. Purpose To evaluate AI prediction variability as an uncertainty quantification (UQ) metric for identifying cases at risk of AI failure in diagnosing cancer at MRI and CT across different cancer types, data sets, and algorithms. Materials and Methods Multicenter data sets and publicly available AI algorithms from three previous studies that evaluated detection of pancreatic cancer on contrast-enhanced CT images, detection of prostate cancer on MRI scans, and prediction of pulmonary nodule malignancy on low-dose CT images were analyzed retrospectively. Each task's algorithm was extended to generate an uncertainty score based on ensemble prediction variability. AI accuracy percentage and partial area under the receiver operating characteristic curve (pAUC) were compared between certain and uncertain patient groups in a range of percentile thresholds (10%-90%) for the uncertainty score using permutation tests for statistical significance. The pulmonary nodule malignancy prediction algorithm was compared with 11 clinical readers for the certain group (CG) and uncertain group (UG). Results In total, 18 022 images were used for training and 838 images were used for testing. AI diagnostic accuracy was higher for the cases in the CG across all tasks (P < .001). At an 80% threshold of certain predictions, accuracy in the CG was 21%-29% higher than in the UG and 4%-6% higher than in the overall test data sets. The lesion-level pAUC in the CG was 0.25-0.39 higher than in the UG and 0.05-0.08 higher than in the overall test data sets (P < .001). For pulmonary nodule malignancy prediction, accuracy of AI was on par with clinicians for cases in the CG (AI results vs clinician results, 80% [95% CI: 76, 85] vs 78% [95% CI: 70, 87]; P = .07) but worse for cases in the UG (AI results vs clinician results, 50% [95% CI: 37, 64] vs 68% [95% CI: 60, 76]; P < .001). Conclusion An AI-prediction UQ metric consistently identified reduced performance of AI in cancer diagnosis. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Babyn in this issue.
引用
- Alves N, Bosma JS, Venkadesh KV, et al. Prediction Variability to Identify Reduced AI Performance in Cancer Diagnosis at MRI and CT. Radiology. 2023. PMID: 37724961. doi:10.1148/radiol.230275
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。