定量评估大型语言模型以简化放射学报告印象:一项多模态回顾性分析
该回顾性比较分析使用MIMIC-IV数据库中750条匿名放射学报告印象,覆盖MRI、CT、US、X线摄影和乳腺X线摄影,比较ChatGPT-3.5、ChatGPT-4、Bard(现称Gemini)和Bing四种公开可用大语言模型在三种提示下简化报告印象的可读性。结果显示,所有四种模型在所有提示下均能简化放射学报告印象(P < .001),但不同模型间存在差异,且提供患者背景或要求七年级水平简化可降低输出阅读年级水平(ChatGPT-4从提示1到提示2除外)(P < .001)。
要点
- 四种大语言模型在所有测试提示下均能简化放射学报告印象(P < .001)。
- 提供患者背景或要求七年级水平简化可降低输出阅读年级水平(P < .001)。
- 不同大语言模型间的简化效果存在差异,具体取决于提示措辞。
核心结论
- 尽管每种大语言模型的成功取决于具体提示措辞,但所有四种模型在所有模态和提示下均简化了放射学报告印象。
研究设计与样本
- 回顾性比较分析,使用MIMIC-IV数据库收集750条匿名放射学报告印象,覆盖MRI、CT、US、X线摄影和乳腺X线摄影及多个解剖区域。
- 比较ChatGPT-3.5、ChatGPT-4、Bard(现称Gemini)和Bing四种公开可用大语言模型(访问时间2023年7月23日至7月26日)。
- 采用三种不同提示评估简化能力:提示1「简化这份放射学报告」;提示2「我是患者,请简化这份放射学报告」;提示3「请将这份放射学报告简化至七年级水平」。
- 每个提示后附放射学报告印象并查询一次;主要结局为可读性评分,采用四种既定可读性指数的平均值评估。
- 使用非参数Wilcoxon符号秩检验比较各模型输出的阅读年级水平。
诊断标准与关键数值
- 可读性通过四种既定可读性指数的平均值评估,以阅读年级水平表示。
- 所有四种大语言模型在所有测试提示下均简化了放射学报告印象(P < .001)。
- 同一提示内不同大语言模型间存在差异。
- 提供患者背景或要求七年级水平简化可降低所有模型和提示的输出阅读年级水平(ChatGPT-4从提示1到提示2除外)(P < .001)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background The complex medical terminology of radiology reports may cause confusion or anxiety for patients, especially given increased access to electronic health records. Large language models (LLMs) can potentially simplify radiology report readability. Purpose To compare the performance of four publicly available LLMs (ChatGPT-3.5 and ChatGPT-4, Bard [now known as Gemini], and Bing) in producing simplified radiology report impressions. Materials and Methods In this retrospective comparative analysis of the four LLMs (accessed July 23 to July 26, 2023), the Medical Information Mart for Intensive Care (MIMIC)-IV database was used to gather 750 anonymized radiology report impressions covering a range of imaging modalities (MRI, CT, US, radiography, mammography) and anatomic regions. Three distinct prompts were employed to assess the LLMs' ability to simplify report impressions. The first prompt (prompt 1) was "Simplify this radiology report." The second prompt (prompt 2) was "I am a patient. Simplify this radiology report." The last prompt (prompt 3) was "Simplify this radiology report at the 7th grade level." Each prompt was followed by the radiology report impression and was queried once. The primary outcome was simplification as assessed by readability score. Readability was assessed using the average of four established readability indexes. The nonparametric Wilcoxon signed-rank test was applied to compare reading grade levels across LLM output. Results All four LLMs simplified radiology report impressions across all prompts tested (P < .001). Within prompts, differences were found between LLMs. Providing the context of being a patient or requesting simplification at the seventh-grade level reduced the reading grade level of output for all models and prompts (except prompt 1 to prompt 2 for ChatGPT-4) (P < .001). Conclusion Although the success of each LLM varied depending on the specific prompt wording, all four models simplified radiology report impressions across all modalities and prompts tested. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Rahsepar in this issue.
引用
- Doshi R, Amin KS, Khosla P, et al. Quantitative Evaluation of Large Language Models to Streamline Radiology Report Impressions: A Multimodal Retrospective Analysis. Radiology. 2024. PMID: 38530171. doi:10.1148/radiol.231593
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。