大型语言模型驱动美国放射学会影像AI注册中心自动化监测
本文描述美国放射学会Assess-AI注册中心利用大型语言模型从影像报告中提取临床相关发现的技术工作流。针对9个用例开发提示词,在开发队列中最终提示词与混合报告参考标准标签的一致性为ICH 0.985、PE 0.997。该工作流展示了规模化报告发现提取的可行性,但独立准确性和临床效用尚未确立。
要点
- LLM提示词由数据科学家与亚专科放射科医生协作开发,用于从报告中提取发现。
- 在开发队列中,ICH和PE的提示词与参考标准一致性分别达0.985和0.997。
- 该一致性来自非独立验证集,独立准确性和临床效用尚未确立。
核心结论
- 基于LLM的提取在Assess-AI中实现了可扩展的、以报告为锚点的影像AI性能监测。
研究设计与样本
- 本研究为技术工作流描述,开发了9个用例的LLM提示词,使用AWS Bedrock可用LLM,以调优队列优化,未报告样本量、中心数或设备信息。
诊断标准与关键数值
- 在Stage 2开发队列中,最终提示词与混合报告参考标准标签的一致性为ICH 0.985、PE 0.997,但该队列非独立验证集。
临床应用与阅片要点
- 日常阅片中,LLM提取可辅助从报告文本中自动识别关键发现,用于AI性能监测。
- 需注意提示词一致性来自开发队列,独立验证不足,不宜直接作为临床诊断标准。
局限性
- 开发队列用于提示词优化和标签构建,非独立验证集,独立准确性和临床效用尚未确立。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVE: To describe the technical workflow enabling scalable automated artificial intelligence (AI) monitoring in the first national imaging AI registry, Assess-AI.
MATERIALS AND METHODS: Large language model (LLM) prompts are developed to extract clinically relevant findings from radiology reports through collaboration between data scientists and subspecialty radiologists. Prompts are optimized using tuning cohorts of use case-specific radiology reports and LLMs available through AWS Bedrock. Such cohorts are used to evaluate prompt accuracy and consistency across 10 repeated runs. Report-AI result pairs submitted to the Assess-AI registry for actively monitored use cases are additionally used to further optimize corresponding prompts.
RESULTS: Prompts were developed for nine use cases. In Stage 2 development cohorts, final-prompt agreement with hybrid report-derived reference standard labels was 0.985 for ICH and 0.997 for PE. Because these cohorts informed prompt refinement and label construction, they were not independent validation sets.
DISCUSSION: The workflow demonstrates feasible report-finding extraction at scale; independent accuracy and clinical utility remain unestablished.
CONCLUSION: LLM-based extraction within Assess-AI enables scalable, report-anchored AI performance monitoring in radiology.
引用
- Schmidt K, Brink L, Bhatia N, et al. AI Monitoring AI with LLMs: The American College of Radiology's Imaging AI Registry. J Am Coll Radiol. 2026. PMID: 42692226. doi:10.1016/j.jacr.2026.08.022
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。