文献速递技术与安全CT

比较大型语言模型与文本嵌入模型对影像报告文本性、语义性及关键性变更的自动分类

Comparing large language models and text embedding models for automated classification of textual, semantic, and critical changes in radiology reports.
PubMed 文献 · Eur J Radiol · 2025-07-14

本研究以400份由住院医师起草、主治医师定稿的成人CT报告为对象,比较开源大型语言模型与词差、Levenshtein/Jaccard相似度、文本嵌入等传统方法对报告变更分级的判别能力;Qwen3-235B-A22B零样本提示与人工评估一致性最高(κ=0.822),嵌入模型表现最差。

要点

  • 报告变更按五点量表分级,1.3%的报告含关键性变更。
  • Qwen3-235B-A22B零样本提示κ=0.822(SD 0.031),最接近人工评估。
  • 词差为最佳传统指标(κ=0.732,SD 0.048),与Qwen3差异校正后未达统计学显著(P=0.064)。

核心结论

  • 大型语言模型(如Qwen3-235B-A22B)对影像报告变更临床意义的评估与专家评价呈中度至强一致性。
  • 大型语言模型在多数情形下优于嵌入方法及传统字符串与词方法,并达统计学显著,显示其作为同行复核支持工具的潜力。

研究设计与样本

  • 研究共纳入400份成人CT报告,将住院医师起草的初版与主治医师定稿版本进行比对。
  • 变更按五点量表评分,从无变化到关键性变化;模型性能以二次加权Cohen kappa(κ)、(平衡)准确率、F1、精确率与召回率评估。
  • 评估者间信度极佳(κ=0.990);所测方法间性能差异有统计学意义(P<0.001)。

诊断标准与关键数值

  • 变更分级采用五点量表,范围从无变化至关键性变化。
  • Qwen3-235B-A22B采用零样本提示,κ=0.822(SD 0.031)。
  • 最佳传统指标为词差,κ=0.732(SD 0.048);两者差异在未校正事后检验中P=0.038,多重检验校正后P=0.064。
  • 嵌入模型表现逊于大型语言模型与经典方法,多数情形差异有统计学意义。

临床应用与阅片要点

  • 作者认为大型语言模型与专家对报告变更临床意义的评价呈中度至强一致性,显示其作为同行复核支持工具的潜力。
  • 原文未据此给出具体阅片或报告流程的推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: Radiology reports can change during workflows, especially when residents draft preliminary versions that attending physicians finalize. We explored how large language models (LLMs) and embedding techniques can categorize these changes into textual, semantic, or clinically actionable types.

METHODS: We evaluated 400 adult CT reports drafted by residents against finalized versions by attending physicians. Changes were rated on a five-point scale from no changes to critical ones. We examined open-source LLMs alongside traditional metrics like normalized word differences, Levenshtein and Jaccard similarity, and text embedding similarity. Model performance was assessed using quadratic weighted Cohen's kappa (κ), (balanced) accuracy, F1, precision, and recall.

RESULTS: Inter-rater reliability among evaluators was excellent (κ = 0.990). Of the reports analyzed, 1.3 % contained critical changes. The tested methods showed significant performance differences (P < 0.001). The Qwen3-235B-A22B model using a zero-shot prompt, most closely aligned with human assessments of changes in clinical reports, achieving a κ of 0.822 (SD 0.031). The best conventional metric, word difference, had a κ of 0.732 (SD 0.048), the difference between the two showed statistical significance in unadjusted post-hoc tests (P = 0.038) but lost significance after adjusting for multiple testing (P = 0.064). Embedding models underperformed compared to LLMs and classical methods, showing statistical significance in most cases.

CONCLUSION: Large language models like Qwen3-235B-A22B demonstrated moderate to strong alignment with expert evaluations of the clinical significance of changes in radiology reports. LLMs outperformed embedding methods and traditional string and word approaches, achieving statistical significance in most instances. This demonstrates their potential as tools to support peer review.

来源:Eur J Radiol,查看原文。版权归原作者所有。

引用

  1. Lindholz M, Burdenski A, Ruppel R, et al. Comparing large language models and text embedding models for automated classification of textual, semantic, and critical changes in radiology reports. Eur J Radiol. 2025. PMID: 40674943. doi:10.1016/j.ejrad.2025.112316

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。