商用人工智能软件在X线肺结节检测与骨龄预测中的性能比较
本研究为Project AIR的一部分,在七家荷兰医院的数据上独立验证了9款商用人工智能产品:2款骨龄预测算法在326例手部X线片上与参考标准强相关(r=0.99),与阅片者相比RMSE无差异;7款肺结节检测算法在386例胸部X线片上,其中4款AUC优于人类阅片者(平均AUC 0.81)。
要点
- 肺结节检测:4款AI算法AUC(0.86-0.93)优于人类阅片者平均AUC 0.81,其余算法与阅片者无差异证据。
- 骨龄预测:2款算法与参考标准强相关(r=0.99),RMSE与人类阅片者无差异。
- 参考标准:骨龄由3至5名专家阅片者设定,肺结节由胸部放射科医师基于CT设定。
核心结论
- 与人类阅片者相比,4款AI算法在胸部X线片肺结节检测中表现更优,其余受测算法未显示性能差异的证据。
研究设计与样本
- 回顾性研究,作为Project AIR的一部分开展,17款合格AI产品中9款在七家荷兰医院的数据上验证。
- 骨龄预测:2款算法,手部X线片来自2017年1月至2022年1月的326例患者(平均年龄10岁±4[SD];173例女性),参考标准由3至5名专家阅片者设定。
- 肺结节检测:7款算法,胸部X线片来自2012年1月至2022年5月的386例患者(平均年龄64岁±11;223例男性),参考标准由胸部放射科医师基于CT设定。
- 随机子集:手部X线片n=95由14名、胸部X线片n=140由17名经验不等的人类阅片者阅读。
诊断标准与关键数值
- 骨龄预测:2款算法与参考标准强相关(r=0.99;两者P<.001)。
- 骨龄预测RMSE:算法1为0.63年(95%CI:0.58,0.69),算法2为0.57年(95%CI:0.52,0.61),阅片者为0.68年(95%CI:0.64,0.73),算法与阅片者之间未见RMSE差异。
- 肺结节检测:人类阅片者平均AUC为0.81(95%CI:0.77,0.85)。
- 肺结节检测:4款算法表现更优,AUC范围0.86-0.93(P值范围<.001至.04)。
临床应用与阅片要点
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Multiple commercial artificial intelligence (AI) products exist for assessing radiographs; however, comparable performance data for these algorithms are limited. Purpose To perform an independent, stand-alone validation of commercially available AI products for bone age prediction based on hand radiographs and lung nodule detection on chest radiographs. Materials and Methods This retrospective study was carried out as part of Project AIR. Nine of 17 eligible AI products were validated on data from seven Dutch hospitals. For bone age prediction, the root mean square error (RMSE) and Pearson correlation coefficient were computed. The reference standard was set by three to five expert readers. For lung nodule detection, the area under the receiver operating characteristic curve (AUC) was computed. The reference standard was set by a chest radiologist based on CT. Randomized subsets of hand (n = 95) and chest (n = 140) radiographs were read by 14 and 17 human readers, respectively, with varying experience. Results Two bone age prediction algorithms were tested on hand radiographs (from January 2017 to January 2022) in 326 patients (mean age, 10 years ± 4 [SD]; 173 female patients) and correlated strongly with the reference standard (r = 0.99; P < .001 for both). No difference in RMSE was observed between algorithms (0.63 years [95% CI: 0.58, 0.69] and 0.57 years [95% CI: 0.52, 0.61]) and readers (0.68 years [95% CI: 0.64, 0.73]). Seven lung nodule detection algorithms were validated on chest radiographs (from January 2012 to May 2022) in 386 patients (mean age, 64 years ± 11; 223 male patients). Compared with readers (mean AUC, 0.81 [95% CI: 0.77, 0.85]), four algorithms performed better (AUC range, 0.86-0.93; P value range, <.001 to .04). Conclusions Compared with human readers, four AI algorithms for detecting lung nodules on chest radiographs showed improved performance, whereas the remaining algorithms tested showed no evidence of a difference in performance. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Omoumi and Richiardi in this issue.
引用
- van Leeuwen KG, Schalekamp S, Rutten MJCM, et al. Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction. Radiology. 2024. PMID: 38193833. doi:10.1148/radiol.230981
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。