文献速递胸部神经骨肌CT

利用大语言模型提取金标准标签验证放射学人工智能模型在新装光子计数CT图像上的性能

Validating Radiology Artificial Intelligence Model Performance on Photon-Counting CT Images Using Large Language Models for Ground Truth Extraction.
PubMed 文献 · J Am Coll Radiol · 2025-11-19

本研究回顾性分析四种FDA cleared的深度学习计算机辅助检测与分诊工具(针对肺栓塞、颅内出血、颈椎骨折、椎体压缩骨折),用大语言模型Llama 3.3从放射学报告中提取二分类金标准标签,比较AI输出以评估性能。结果显示光子计数CT与非光子计数CT队列间性能无统计学显著差异,LLM标签与最终人工标注的一致性(κ=0.731)与阅片者间一致性(κ=0.720)相当。结论认为LLM可自动化提取金标准标签,支持AI工具的快速本地验证。

要点

  • LLM(Llama 3.3)从放射学报告提取的二分类标签与人工标注一致性κ=0.731,与阅片者间一致性κ=0.720相当。
  • 四种AI工具在光子计数CT与非光子计数CT队列间性能无统计学显著差异。
  • LLM提取标签可实现四种诊断任务的快速性能评估,支持新硬件输入漂移下的本地验证。

核心结论

  • 大语言模型可从放射学报告中自动化提取金标准标签,为人工标注提供可扩展且高效的替代方案,支持AI工具的快速本地验证。

研究设计与样本

  • 回顾性分析四种FDA cleared的深度学习计算机辅助检测与分诊工具,靶向肺栓塞、颅内出血、颈椎骨折和椎体压缩骨折。

诊断标准与关键数值

  • LLM提取标签与最终人工标注的一致性κ=0.731,阅片者间一致性κ=0.720。
  • 光子计数CT与非光子计数CT队列间AI工具性能无统计学显著差异。

临床应用与阅片要点

  • LLM可从放射学报告自动提取二分类金标准标签,用于AI工具性能的快速评估与监测。
  • 该方法可支持新成像硬件(如光子计数CT)输入漂移下的AI工具本地验证。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: The aim of this study was to evaluate the feasibility of using large language models (LLMs) to automate ground truth label extraction from radiology reports, enabling scalable assessment and monitoring of radiologic artificial intelligence (AI) tools. The framework was tested by validating AI model performance on a newly installed photon-counting CT (PCCT) scanner.

METHODS: Four FDA-cleared deep learning-based computer-aided detection and triage tools targeting pulmonary embolism, intracranial hemorrhage, cervical spinal fractures, and vertebral compression fractures were retrospectively analyzed. Radiology reports from examinations acquired using the new PCCT scanner and conventional scanners were processed using an LLM (Llama 3.3) to extract binary ground truth labels. AI outputs were compared with these labels to estimate performance metrics. Discrepant cases were adjudicated by three human annotators, with interrater reliability measured using Fleiss's κ test. Performance metrics were recalculated after partial human correction of LLM errors.

RESULTS: LLM-extracted labels enabled rapid performance assessment across all four diagnostic tasks. There were no statistically significant differences in performance between the PCCT and non-PCCT cohorts. In discrepant cases, the agreement between LLM labels and final human annotations (κ = 0.731) was comparable with interreader agreement (κ = 0.720), supporting the reliability of LLM labeling.

CONCLUSIONS: LLMs can be used to automate ground truth label extraction from radiology reports, offering a scalable and efficient alternative to manual annotation. This method supports rapid local validation of AI tools, even in response to input drift from new imaging hardware.

来源:J Am Coll Radiol,查看原文。版权归原作者所有。

引用

  1. Ng YS, Kanani MM, King WE, et al. Validating Radiology Artificial Intelligence Model Performance on Photon-Counting CT Images Using Large Language Models for Ground Truth Extraction. J Am Coll Radiol. 2025. PMID: 41265766. doi:10.1016/j.jacr.2025.11.024

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。