基于乳腺X线摄影筛查个人表现方案的乳腺癌检测人工智能算法性能评估
本研究比较了人工阅片者与商用人工智能算法在PERFORMS测试集上的诊断性能。552名阅片者与AI算法分别解读两个各含60例挑战性病例的测试集,AI在开发者建议召回阈值下特异度高于人工阅片者(89% vs 76%,P=0.003),敏感度无显著差异;匹配人工平均表现阈值时两者性能相当。AI诊断性能与平均人工阅片者相当。
要点
- AI算法在开发者建议阈值下特异度89%高于人工阅片者76%(P=0.003),敏感度84%与90%无显著差异(P=0.34)。
- AI与人工阅片者的AUC分别为0.93和0.88,无显著差异(P=0.15)。
核心结论
- 在评估PERFORMS方案的两个富集测试集时,AI的诊断性能与平均人工阅片者相当。
研究设计与样本
- 回顾性研究,两个PERFORMS测试集各含60例挑战性病例,共161个正常乳腺、70个恶性乳腺和9个良性乳腺,由552名人工阅片者于2018年5月至2021年3月间解读,AI算法于2022年评估,AI对每侧乳腺分别赋予恶性可疑度评分并以每侧乳腺最高分进行性能评估。
临床应用与阅片要点
- 原文未据此给出推荐。
局限性
- 由于测试集大小限制,无法证明等效性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background The Personal Performance in Mammographic Screening (PERFORMS) scheme is used to assess reader performance. Whether this scheme can assess the performance of artificial intelligence (AI) algorithms is unknown. Purpose To compare the performance of human readers and a commercially available AI algorithm interpreting PERFORMS test sets. Materials and Methods In this retrospective study, two PERFORMS test sets, each consisting of 60 challenging cases, were evaluated by human readers between May 2018 and March 2021 and were evaluated by an AI algorithm in 2022. AI considered each breast separately, assigning a suspicion of malignancy score to features detected. Performance was assessed using the highest score per breast. Performance metrics, including sensitivity, specificity, and area under the receiver operating characteristic curve (AUC), were calculated for AI and humans. The study was powered to detect a medium-sized effect (odds ratio, 3.5 or 0.29) for sensitivity. Results A total of 552 human readers interpreted both PERFORMS test sets, consisting of 161 normal breasts, 70 malignant breasts, and nine benign breasts. No difference was observed at the breast level between the AUC for AI and the AUC for human readers (0.93% and 0.88%, respectively; P = .15). When using the developer's suggested recall score threshold, no difference was observed for AI versus human reader sensitivity (84% and 90%, respectively; P = .34), but the specificity of AI was higher (89%) than that of the human readers (76%, P = .003). However, it was not possible to demonstrate equivalence due to the size of the test sets. When using recall thresholds to match mean human reader performance (90% sensitivity, 76% specificity), AI showed no differences inperformance, with a sensitivity of 91% (P =. 73) and a specificity of 77% (P = .85). Conclusion Diagnostic performance of AI was comparable with that of the average human reader when evaluating cases from two enriched test sets from the PERFORMS scheme. © RSNA, 2023 See also the editorial by Philpotts in this issue.
引用
- Chen Y, Taib AG, Darker IT, et al. Performance of a Breast Cancer Detection AI Algorithm Using the Personal Performance in Mammographic Screening Scheme. Radiology. 2023. PMID: 37668522. doi:10.1148/radiol.223299
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。