专用大语言模型与GPT-4o在CT和MRI影像报告摘要生成中的性能比较
本研究基于某医院回顾性收集的影像报告开发了专用报告摘要大语言模型LLM-RadSum,并在内部测试集(106 247份)及四家其他医院的外部测试集(17 091份)上评估其最长公共子序列F1分数。由三名高级放射科医师和两名临床医师对1800份报告进行人工评价,结果显示LLM-RadSum在事实一致性、印象连贯性、医疗安全性和临床可用性方面达标率超过81.5%,而GPT-4o至少27.8%的输出需调整;LLM-RadSum整体中位F1分数高于GPT-4o(0.58 vs 0.30,P < .001)。
要点
- LLM-RadSum在1800份人工评价集中位F1分数为0.58,GPT-4o为0.30(P < .001)。
- LLM-RadSum超过81.5%(1467/1800)的输出在事实一致性、印象连贯性、医疗安全性和临床可用性方面达标。
- GPT-4o至少27.8%(501/1800)的输出在上述方面需要调整。
核心结论
- 专用报告摘要大语言模型LLM-RadSum在生成影像报告摘要方面性能优于通用大语言模型GPT-4o。
研究设计与样本
- 回顾性收集某医院报告按9:1分为训练集(956 219份)和内部测试集(106 247份),并在四家其他医院的外部测试集(17 091份)上评估;仅纳入包含表现和印象部分的CT和MRI报告;人工评价集从内部和外部测试集中随机抽取1800份报告,覆盖CT和MRI及胸部、颈部、头部、骨盆、腹部、乳腺等解剖部位,由三名高级放射科医师和两名临床医师评估;采用t检验比较两种模型的F1分数。
诊断标准与关键数值
- 以基于最长公共子序列的F1分数评价报告摘要质量;LLM-RadSum在内部测试集中位F1为0.75,外部测试集为0.44,人工评价集(n=1800)为0.58;LLM-RadSum在人工评价集中位F1为0.58(IQR 0.44-0.77),GPT-4o为0.30(IQR 0.23-0.37),P < .001;LLM-RadSum超过81.5%(1467/1800)的输出在事实一致性、印象连贯性、医疗安全性和临床可用性方面达到高级放射科医师和临床医师标准,GPT-4o至少27.8%(501/1800)的输出需在这些方面调整;LLM-RadSum在解剖区域、检查方式、性别、年龄和印象长度方面均表现更优(均P < .001)。
临床应用与阅片要点
- 专用大语言模型LLM-RadSum生成的CT和MRI报告摘要在事实一致性、印象连贯性、医疗安全性和临床可用性方面达标率高于GPT-4o,GPT-4o输出中需调整的比例至少为27.8%。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Although the general-purpose large language model (LLM) GPT-4o (OpenAI) has shown promise in radiology language processing, it remains unclear whether the performance of GPT-4o in report summarization is better than that of an LLM specifically designed for this task. Purpose To compare the performance of a specialized LLM with that of GPT-4o in the comprehensive summarization of radiology reports. Materials and Methods A specialized LLM for report summarization (LLM-RadSum) was developed using retrospectively collected reports from a hospital, divided into training and internal test sets (9:1 ratio). The F1 scores based on the longest common subsequences were evaluated on the internal test set and an external test set of reports from four other hospitals. Only CT and MRI reports containing findings and impressions sections were included. For comparison with GPT-4o, a human evaluation set included 1800 reports randomly selected from the internal and external test sets, ensuring balanced coverage of imaging modalities (CT, MRI) and anatomic sites (chest, neck, head, pelvis, abdomen, breast). Three senior radiologists and two clinicians assessed this set, focusing on factual consistency, impression coherence, medical safety, and clinical use. A t test was performed to compare F1 scores between models. Results The training, internal test, and external test sets were composed of 956 219, 106 247, and 17 091 reports, respectively. The developed LLM-RadSum achieved median F1 scores for report summarization of 0.75 and 0.44 on the internal and external test sets and 0.58 on the human evaluation set (n = 1800). More than 81.5% (1467 of 1800) of outputs from LLM-RadSum met the standards of senior radiologists and clinicians regarding factual consistency, impression coherence, medical safety, and clinical use. In contrast, at least 27.8% (501 of 1800) of outputs from GPT-4o required adjustments in these aspects. Overall, LLM-RadSum achieved a higher median F1 score for report summarization compared with GPT-4o (0.58 [IQR: 0.44-0.77] vs 0.30 [IQR: 0.23-0.37]; P < .001); superior performance from LLM-RadSum was observed across anatomic regions, modalities, sexes, ages, and impression lengths (all P < .001). Conclusion A specialized LLM for report summarization had better performance than GPT-4o, a general-purpose LLM, in generating radiology report summaries. © RSNA, 2025 Supplemental material is available for this article. See also the editorial by Huisman and Rasoolzadeh in this issue.
引用
- Zheng S, Zhao N, Wang J, et al. Comparison of a Specialized Large Language Model with GPT-4o for CT and MRI Radiology Report Summarization. Radiology. 2025. PMID: 40892451. doi:10.1148/radiol.243774
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。