文献速递神经骨肌技术与安全CTDR

人工智能用于教学病例策展:评估模型在影像报告差异检测中的表现

Artificial Intelligence for Teaching Case Curation: Evaluating Model Performance on Imaging Report Discrepancies.
PubMed 文献 · Acad Radiol · 2025-03-09

回顾性研究纳入2017年1月至2021年12月的非工作时间头部CT和骨肌X线检查,标注受训者初步解读与最终主治报告之间的差异水平(5分制),并微调RadBERT模型检测差异。微调模型总体准确率90.5%、特异度95.5%、敏感度66.3%,敏感度随差异评分升高而提高;模型筛选病例集中全部差异和重大差异的患病率显著高于随机病例集。

要点

  • 微调RadBERT检测报告差异的总体准确率90.5%、特异度95.5%、敏感度66.3%。
  • 敏感度随差异评分升高而提高:评分2为49%(34/70),评分3为67%(47/62),评分4/5为81%(35/43)。
  • 模型筛选集全部差异和重大差异(评分4或5)的患病率均显著高于随机病例集(P<0.05)。

核心结论

  • 大语言模型能够检测受训者报告差异,包括较高和较低评分的差异,并可能改善住院医师教育的病例集策展,同时可作为受训者监督工具。

研究设计与样本

  • 回顾性研究,纳入2017年1月至2021年12月的非工作时间头部CT和骨肌X线检查,对受训者初步解读与最终主治报告之间的差异水平进行5分制标注,微调在大量放射学文本语料上预训练的RadBERT模型用于差异检测,并与Mixstral 8×7B、Mistral 7B和Llama2比较,使用随机选择的留出测试集评估性能,采用F1统计量进行模型比较,采用Pearson卡方检验评估组间差异患病率和评分(显著性设为p<0.05)。

诊断标准与关键数值

  • 微调大语言模型检测差异的总体准确率90.5%,特异度95.5%,敏感度66.3%。
  • 模型敏感度随差异评分升高而提高:评分2为49%(34/70),评分3为67%(47/62),评分4/5为81%(35/43),评分3和评分4/5与评分2相比p<0.05。
  • 模型筛选集全部差异和重大差异(评分4或5)的患病率均显著高于随机病例集(两者P<0.05)。

临床应用与阅片要点

  • 作者认为大语言模型可检测受训者报告差异,包括较高和较低评分的差异,并可能改善住院医师教育的病例集策展,同时可作为受训者监督工具。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: Assess the feasibility of using a large language model (LLM) to identify valuable radiology teaching cases through report discrepancy detection.

MATERIALS AND METHODS: Retrospective study included after-hours head CT and musculoskeletal radiograph exams from January 2017 to December 2021. Discrepancy level between trainee's preliminary interpretation and final attending report was annotated on a 5-point scale. RadBERT, an LLM pretrained on a vast corpus of radiology text, was fine-tuned for discrepancy detection. For comparison and to ensure the robustness of the approach, Mixstral 8×7B, Mistral 7B, and Llama2 were also evaluated. The model's performance in detecting discrepancies was evaluated using a randomly selected hold-out test set. A subset of discrepant cases identified by the LLM was compared to a random case set by recording clinical parameters, discrepant pathology, and evaluating possible educational value. F1 statistic was used for model comparison. Pearson's chi-squared test was employed to assess discrepancy prevalence and score between groups (significance set at p<0.05).

RESULTS: The fine-tuned LLM model achieved an overall accuracy of 90.5% with a specificity of 95.5% and a sensitivity of 66.3% for discrepancy detection. The model sensitivity significantly improved with higher discrepancy scores, 49% (34/70) for score 2 versus 67% (47/62) for score 3, and 81% (35/43) for score 4/5 (p<0.05 compared to score 2). LLM-curated set showed a significant increase in the prevalence of all discrepancies and major discrepancies (scores 4 or 5) compared to a random case set (P<0.05 for both). Evaluation of the clinical characteristics from both the random and discrepant case sets demonstrated a broad mix of pathologies and discrepancy types.

CONCLUSION: An LLM can detect trainee report discrepancies, including both higher and lower-scoring discrepancies, and may improve case set curation for resident education as well as serve as a trainee oversight tool.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Bartley M, Huemann Z, Hu J, et al. Artificial Intelligence for Teaching Case Curation: Evaluating Model Performance on Imaging Report Discrepancies. Acad Radiol. 2025. PMID: 40058990. doi:10.1016/j.acra.2025.02.011

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。