文献速递神经技术与安全CT

评估人工智能模型的性能:来自美国功能神经放射学会人工智能竞赛的洞见

Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.
PubMed 文献 · AJNR Am J Neuroradiol · 2024-09-09

美国功能神经放射学会组织多中心人工智能竞赛,用5家机构1201例匿名全头颅NCCT、以5名神经放射科医师共识为参考标准,评估模型在病灶检出、年龄相关正常性判断和紧急干预识别三项任务上的表现。4支团队处理1177例,任务1的AUC为0.49–0.59,任务2两支团队AUC为0.57和0.52,任务3与参考标准一致性极低至无。结论指出模型表现普遍不达预期,机构采用前需充分验证。

要点

  • 任务1病灶检出的AUC仅0.49–0.59,接近随机水平。
  • 任务2年龄相关正常性判断仅2支团队完成,AUC为0.57和0.52。
  • 任务3紧急干预识别与参考标准一致性极低至无。

核心结论

  • 首届ASFNR竞赛显示模型表现与预期存在差距,模型评估大多不达标。
  • 随着人工智能工具融入临床工作流,神经放射科医师须仔细认识这些技术的能力、约束和一致性。
  • 机构采用这些算法前,充分验证对确保临床环境中可接受的性能水平至关重要。

研究设计与样本

  • 研究类型为多中心人工智能竞赛,由美国功能神经放射学会组织。
  • 数据集为来自5家机构的1201例匿名全头颅NCCT临床扫描,涵盖正常及急性缺血性卒中、颅内出血、创伤性脑损伤和占位效应等病变。
  • 参考标准为5名神经放射科医师标注的共识判读。
  • 竞赛在线公布,邀请学术机构和公司参加;独立中心分析评估各模型性能。
  • 4支团队处理1177例研究,患者中位年龄62岁,四分位距33岁;19支团队注册,4支提交最终结果,无商业实体参与。

诊断标准与关键数值

  • 任务1病灶检出:各人工智能模型AUC范围为0.49–0.59。
  • 任务2年龄相关正常性评估:两支完成任务的团队AUC分别为0.57和0.52。
  • 任务3紧急干预识别:各团队与参考标准一致性极低至无。
  • 每个模型均生成准确率、敏感度、特异度、阳性预测值、阴性预测值及ROC曲线和AUC。

临床应用与阅片要点

  • 原文结论强调神经放射科医师须仔细认识人工智能技术的能力、约束和一致性。
  • 原文结论强调机构采用这些算法前需充分验证以确保临床环境中可接受的性能水平。

局限性

  • 竞赛无商业实体参与。
  • 19支注册团队中仅4支提交最终结果。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

BACKGROUND AND PURPOSE: Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency.

MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve.

RESULTS: Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth.

CONCLUSIONS: To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.

来源:AJNR Am J Neuroradiol,查看原文。版权归原作者所有。

引用

  1. Jiang B, Ozkara BB, Zhu G, et al. Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition. AJNR Am J Neuroradiol. 2024. PMID: 38663992. doi:10.3174/ajnr.A8317

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。