ACR Assess-AI：临床影像人工智能真实世界性能监测注册系统
本文介绍ACR Assess-AI注册系统的设计、基础设施与功能，该系统通过ACR Connect平台收集去标识化AI输出、报告文本和DICOM元数据，利用大语言模型从报告中提取替代标签并集中计算一致性，通过交互式仪表板反馈结果，支持颅内出血、肺栓塞、气胸、大血管闭塞、骨龄和颈椎骨折等AI用例的真实世界性能监测。
要点
- Assess-AI注册系统用于监测临床影像AI模型的部署后真实世界性能。
- 系统通过大语言模型从放射报告中提取替代标签，集中计算与AI输出的一致性。
- 支持颅内出血、肺栓塞、气胸、大血管闭塞、骨龄和颈椎骨折等AI用例。
核心结论
- Assess-AI为影像AI模型部署后性能监测提供了可扩展且保护隐私的框架。
- 该注册系统通过标准化数据接入、大语言模型替代标签、交互分析和本地裁决弥补了真实世界AI性能评估的关键空白。
研究设计与样本
- Assess-AI是国家放射学数据注册系统内的一个注册系统，参与机构通过ACR Connect平台提交去标识化AI输出、放射报告文本和DICOM研究元数据。
- 数据经标准化后，使用基于大语言模型的提示流程从放射报告中提取替代标签，与AI输出进行比较并集中计算一致性。
- 结果通过交互式仪表板交付，机构可通过Forensics App本地重新识别研究以进行质量审查。
诊断标准与关键数值
- 系统支持颅内出血、肺栓塞、气胸、大血管闭塞、骨龄和颈椎骨折等影像AI用例。
- 参与机构可查看数据完整性、监测纵向一致性趋势、与注册基准比较性能，并按人口学或技术因素探索不一致性。
临床应用与阅片要点
- 日常阅片和报告中可利用该注册系统监测AI模型在本地部署后的真实世界性能。
- 通过本地审查工作流对不一致病例进行详细评估，支持根因分析和AI治理。
局限性
- 系统当前支持的AI用例有限，正在扩展以纳入更多模态、模型类型和风险调整基准。
- 替代标签从放射报告中提取，可能受报告质量影响。
以上解读依据论文摘要（全文未开放获取或许可不允许翻译转载）。
英文原文（摘要）· 点击展开
OBJECTIVE: To describe the design, infrastructure, and functionality of the ACR's Assess-AI registry, a national quality registry created to monitor the real-world performance of clinical imaging artificial intelligence (AI) models.
METHODS: Assess-AI is a registry within the National Radiology Data Registry that enables participating facilities to submit de-identified AI output, radiology report text, and DICOM study metadata through the ACR Connect platform. Data are normalized and compared with surrogate labels extracted from radiology reports using large language model-based prompting pipelines. Concordance between AI outputs and extracted surrogate labels is computed centrally, with results delivered through interactive dashboards. Facilities may locally re-identify studies via the Forensics App for quality review.
RESULTS: Assess-AI currently supports multiple imaging AI use cases including intracranial hemorrhage, pulmonary embolism, pneumothorax, large-vessel occlusion, bone age, and cervical spine fracture. Participating facilities can visualize data completeness, monitor longitudinal concordance trends, compare performance with registry benchmarks, and explore discordance by demographic or technical factors. Local review workflows allow detailed evaluation of discordant cases, supporting root-cause analysis and AI governance.
DISCUSSION: Assess-AI provides a scalable, privacy-preserving framework for postdeployment performance monitoring of imaging AI models. By combining standardized data ingestion, large language model-based surrogate labeling, interactive analytics, and local adjudication, the registry addresses critical gaps in evaluating real-world AI performance. Ongoing expansion will incorporate additional modalities, model types, and risk-adjusted benchmarking to further enhance clinical utility.
引用
- Coombs LP, Brink L, Kim W, et al. ACR's Assess-AI: A Registry for Real-World Performance Monitoring of Clinical Imaging Artificial Intelligence. J Am Coll Radiol. 2026. PMID: 42066927. doi:10.1016/j.jacr.2026.04.024
供执业医师工作参考，不替代临床判断；数值与分级以指南原文为准。