文献速递技术与安全

预测放射学人工智能应用的价值:对一组模型的大规模部署前评估

Predicting the Value of Radiology Artificial Intelligence Applications: Large-Scale Predeployment Evaluation of a Portfolio of Models.
PubMed 文献 · AJR Am J Roentgenol · 2026-03-04

本研究于2022年7月至2024年11月在一家大型全国性放射学机构开展前瞻性研究,对单一供应商的13个AI模型(覆盖12项临床任务)进行部署前评估,由四名放射科医生工作组提出固有价值属性并加权评分,结合88,645例检查的传统与增强检测指标预测模型整体价值,部署后问卷调查显示12项任务中10项的价值感知与预测一致。

要点

  • 部署前评估方法结合任务固有价值评估与放射科医生及AI性能指标,可预测AI模型整体价值。
  • 放射科医生通常具有更高的PPV,而AI通常具有更高的敏感度。
  • 模型增强检测率绝对值为0.03-2.27%,相对值为4.5-60.5%,各模型差异很大。

核心结论

  • 提出一种结构化的AI模型部署前评估方法,将任务固有价值评估与放射科医生和AI性能指标相结合,验证调查显示部署前预测与部署后实际价值感知高度一致。

研究设计与样本

  • 前瞻性研究,2022年7月至2024年11月在一家大型全国性放射学机构开展,对单一供应商的13个AI模型(覆盖12项临床任务)进行部署前评估,由四名放射科医生工作组识别价值属性并加权评分,基于临床报告评估放射科医生与AI在88,645例检查上的性能,结合传统指标与增强检测指标预测模型整体价值,部署后由放射科医生完成问卷调查。

诊断标准与关键数值

  • 工作组识别出三个最可能贡献AI辅助固有价值的属性:任务的繁琐程度、放射科医生遗漏发现的可能性、遗漏发现的潜在临床影响;12项任务中5项、5项、2项分别被评为高、中、低固有价值;模型增强检测率绝对值为0.03-2.27%,相对值为4.5-60.5%;13个模型中5个、5个、3个分别预测为高、中、低整体价值;问卷应答率43.2%(54/125),12项任务中10项的价值感知类别与工作组预测一致。

临床应用与阅片要点

  • 该实用评估方法可帮助放射学机构在放射学AI模型的循证采购和部署决策中提供指导。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

BACKGROUND. Real-world performance of radiology artificial intelligence (AI) applications frequently diverges from previously reported results, creating challenges in anticipating a model's clinical value and impact. OBJECTIVE. The purpose of this study was to develop a structured predeployment evaluation method for radiology AI models that combines standard performance metrics with new augmentation metrics in predicting the overall value of an AI model and to test this method's predictions against radiologists' real-world postdeployment perceptions of model value. METHODS. In this prospective study, from July 2022 to November 2024, a large national radiology practice conducted a predeployment evaluation of a single vendor's portfolio of 13 AI models for 12 clinical tasks. A four-radiologist workgroup identified attributes contributing to the inherent value of AI assistance for clinical tasks, assigned weights to those attributes, and rated models accordingly. Performance of radiologists (based on clinical reports) and AI was assessed for 88,645 examinations across clinical sites by use of conventional metrics and augmentation metrics reflecting enhanced-detection cases (i.e., AI-detected radiologist-missed positive cases). The workgroup combined inherent task values and pooled AI performance to predict models' overall value. Radiologists completed a postdeployment survey. RESULTS. The workgroup identified three attributes most likely to contribute to the inherent value of AI assistance: the tediousness of the task, the likelihood that the radiologist would miss the finding, and the potential clinical impact of a missed finding. Five, five, and two tasks were rated as having high, medium, and low inherent value, respectively. Across tasks, radiologists generally had higher PPV, whereas AI generally had higher sensitivity. Models showed widely varying absolute and relative enhanced detection rates (0.03-2.27% and 4.5-60.5%, respectively). Five, five, and three models were predicted to have high, medium, and low overall value, respectively. The survey response rate was 43.2% (54/125). Perceived value categories agreed between survey respondents and workgroup predictions for 10 of 12 tasks. CONCLUSION. We present a structured method for predeployment evaluation of AI models' potential value, combining task-inherent value assessments with radiologist and AI performance metrics. A validation survey indicated high agreement between predeployment predictions and real-world postdeployment value perceptions. CLINICAL IMPACT. This practical evaluation approach can help guide radiology practices in evidence-based purchasing and deployment decisions for radiology AI models.

来源:AJR Am J Roentgenol,查看原文。版权归原作者所有。

引用

  1. Larson DB, Poff JA, Krishnan S, et al. Predicting the Value of Radiology Artificial Intelligence Applications: Large-Scale Predeployment Evaluation of a Portfolio of Models. AJR Am J Roentgenol. 2026. PMID: 41779377. doi:10.2214/AJR.25.34340

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。