文献速递技术与安全CTMRDR乳腺X线

构建大型语言模型以从影像学报告中生成印象

Constructing a Large Language Model to Generate Impressions from Findings in Radiology Reports.
PubMed 文献 · Radiology · 2024-01-01

该研究开发了一个大型语言模型(LLM),用于根据影像学表现生成放射学印象,并在专业性和语言性两个维度评估其性能。在3988例患者的测试集中,LLM生成印象的中位召回率、精确率和F1分数分别为0.775、0.84和0.772;在1014例患者的子集中,专家小组对LLM生成印象的总体中位评分为5分(满分5分)。结论认为,该LLM生成的放射学印象在专业性和语言性上适用于全谱系放射学检查。

要点

  • LLM生成印象的中位召回率、精确率和F1分数分别为0.775、0.84和0.772(以最终印象为参考标准)。
  • 在1014例子集中,专家小组对LLM生成印象的总体中位评分为5分(IQR 5-5)。
  • 该LLM基于20 GB医学和通用文本数据预训练,并使用1.5 GB微调数据集(含800份放射学报告)进行微调。

核心结论

  • 开发的LLM生成的放射学印象在专业性和语言性上适用于全谱系放射学检查。

研究设计与样本

  • 回顾性研究,于2023年8月2日至31日在上海市第一人民医院由6名放射科医生记录影像检查表现并使用LLM辅助生成报告印象,涵盖CT、MRI、X线摄影、乳腺X线摄影等多种模态及多个解剖部位;LLM使用20 GB医学和通用文本数据预训练,微调数据集为1.5 GB(含800份放射学报告及配对指令和输出);测试集2包括3988例患者(中位年龄56岁,IQR 40-68岁;男性2159例),以最终印象为参考标准。

诊断标准与关键数值

  • 以最终放射学印象为参考标准,LLM生成印象的中位召回率为0.775(IQR 0.56-1),中位精确率为0.84(IQR 0.611-1),中位F1分数为0.772(IQR 0.578-0.957);在1014例患者子集中,专家小组基于科学术语、连贯性、具体诊断、鉴别诊断、管理建议、正确性、全面性、无害性和无偏倚性采用五点Likert量表(5=强烈同意)评分,总体中位评分为5分(IQR 5-5),范围从4分(IQR 3-5)到5分(IQR 5-5)。

临床应用与阅片要点

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

Background The specialization and complexity of radiology makes the automatic generation of radiologic impressions (ie, a diagnosis with differential diagnosis and management recommendations) challenging. Purpose To develop a large language model (LLM) that generates impressions based on imaging findings and to evaluate its performance in professional and linguistic dimensions. Materials and Methods Six radiologists recorded imaging examination findings from August 2 to 31, 2023, at Shanghai General Hospital and used the developed LLM before routinely writing report impressions for multiple radiologic modalities (CT, MRI, radiography, mammography) and anatomic sites (cranium and face, neck, chest, upper abdomen, lower abdomen, vessels, bone and joint, spine, breast), making necessary corrections and completing the radiologic impression. A subset was defined to investigate cases where the LLM-generated impressions differed from the final radiologist impressions by excluding identical and highly similar cases. An expert panel scored the LLM-generated impressions on a five-point Likert scale (5 = strongly agree) based on scientific terminology, coherence, specific diagnosis, differential diagnosis, management recommendations, correctness, comprehensiveness, harmlessness, and lack of bias. Results In this retrospective study, an LLM was pretrained using 20 GB of medical and general-purpose text data. The fine-tuning data set comprised 1.5 GB of data, including 800 radiology reports with paired instructions (describing the output task in natural language) and outputs. Test set 2 included data from 3988 patients (median age, 56 years [IQR, 40-68 years]; 2159 male). The median recall, precision, and F1 score of LLM-generated impressions were 0.775 (IQR, 0.56-1), 0.84 (IQR, 0.611-1), and 0.772 (IQR, 0.578-0.957), respectively, using the final impressions as the reference standard. In a subset of 1014 patients (median age, 57 years [IQR, 42-69 years]; 528 male), the overall median expert panel score for LLM-generated impressions was 5 (IQR, 5-5), ranging from 4 (IQR, 3-5) to 5 (IQR, 5-5). Conclusion The developed LLM generated radiologic impressions that were professionally and linguistically appropriate for a full spectrum of radiology examinations. © RSNA, 2024 Supplemental material is available for this article.

来源:Radiology,查看原文。版权归原作者所有。

引用

  1. Zhang L, Liu M, Wang L, et al. Constructing a Large Language Model to Generate Impressions from Findings in Radiology Reports. Radiology. 2024. PMID: 39287525. doi:10.1148/radiol.240885

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。