使用大语言模型对神经影像学报告进行客观质量评估
本研究分析277份由10位放射科医生撰写的神经影像学报告,由另一位放射科医生按六项质量标准评估,并测试三种本地部署的大语言模型(LLaMA 3.2、DeepSeek R1:7B、Gemma 3:4B)的分类能力。模型在识别阳性类别上表现较强,DeepSeek在报告结构(70.40%)和附加检查建议(68.59%)准确率最高,但在检测报告完整性或诊断差异等问题上F1分数较低。作者认为大语言模型可能辅助自动化报告质量控制的部分环节,但识别细微错误或复杂临床细微差别仍有局限。
要点
- 大语言模型在识别报告结构合格和附加检查建议等阳性类别上准确率较高,DeepSeek分别达70.40%和68.59%。
- 大语言模型检测报告完整性缺陷和诊断差异等问题的能力下降,相应类别F1分数低。
- 当前模型识别细微错误或复杂临床细微差别仍有局限,可能辅助报告质量控制的部分自动化。
核心结论
- 大语言模型可能辅助放射学报告质量控制的部分自动化,但识别细微错误或复杂临床细微差别仍有局限;进一步改进后可能有助于提高一致性和支持临床决策。
研究设计与样本
- 分析277份神经影像学报告,原由10位放射科医生撰写,再由另一位放射科医生评估正式质量和诊断质量。报告按六项质量标准标注:诊断差异、报告结构、清晰度、完整性、分级/分期系统使用、附加检查建议。评估三种本地部署的大语言模型(LLaMA 3.2、DeepSeek R1:7B、Gemma 3:4B)按这些标准分类报告的能力。
诊断标准与关键数值
- 以另一位放射科医生对六项质量标准的评估为参照,评估三种大语言模型的分类性能。
- DeepSeek在报告结构分类中准确率最高,为70.40%。
- DeepSeek在附加检查建议分类中准确率最高,为68.59%。
- 检测报告完整性或诊断差异等问题时性能显著下降,相应类别F1分数低。
临床应用与阅片要点
- 大语言模型可能辅助自动化放射学报告质量控制的部分环节。
- 当前模型识别细微错误或复杂临床细微差别仍有局限,进一步改进后可能有助于提高一致性和支持临床决策。
局限性
- 当前模型在识别细微错误或复杂临床细微差别方面存在局限。
- 模型在检测报告完整性或诊断差异等问题上性能下降,相应类别F1分数低。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
AIM: Radiology reports are essential for clinical decision-making and must meet standards of clarity, completeness, and diagnostic accuracy. However, quality assessment is often subjective, time-consuming, and dependent on expert reviewers. Large language models (LLMs) offer a promising alternative for automating this process. This study evaluates whether LLMs can objectively and consistently assess the formal and diagnostic quality of neuroradiology reports.
MATERIALS AND METHODS: We analysed 277 neuroradiology reports, originally authored by 10 radiologists and subsequently evaluated by a different radiologist to assess formal and diagnostic quality. Reports were annotated using six quality criteria: diagnostic discrepancy, report structure, clarity, completeness, grading/staging system use, and recommendation of additional tests. Three locally deployed LLMs (LLaMA 3.2, DeepSeek R1:7B, and Gemma 3:4B) were evaluated for their ability to classify reports according to these criteria.
RESULTS: LLMs showed strong performance in recognising positive categories, with DeepSeek achieving the highest accuracy in report structure (70.40%) and test recommendations (68.59%). However, performance declined significantly in detecting issues such as report completeness or diagnostic discrepancies, reflected in low F1-scores for those categories.
CONCLUSION: While current models have limitations in identifying subtle errors or complex clinical nuances, LLMs may assist in automating aspects of radiology report quality control. With further refinement, they could help improve consistency and support clinical decision-making.
引用
- López-Úbeda P, Martín-Noguerol T, Luna A, et al. Objective quality assessment of neuroradiology reports using large language models. Clin Radiol. 2026. PMID: 41831341. doi:10.1016/j.crad.2026.107291
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。