文献速递介入技术与安全

大语言模型用于简化介入放射学报告:比较分析

Large Language Models for Simplified Interventional Radiology Reports: A Comparative Analysis.
PubMed 文献 · Acad Radiol · 2024-09-30

本研究对109份介入放射学报告,用专有模型(GPT-4、GPT-3.5 Turbo、Claude-3-Opus、Gemini Ultra)和开源模型(Mistral-7b、Mistral-8×7b)进行简化,从六个定性指标和四项可读性指标评估。GPT-4与Claude-3-Opus定性表现无显著差异且优于其他模型,GPT-4可读性优于Claude-3-Opus;所有模型均出现信任破坏性错误。

要点

  • GPT-4与Claude-3-Opus在六项定性指标上无显著差异(Bonferroni校正后P=1),且优于其他评估模型。
  • GPT-4内容错误和信任破坏性错误最少,Claude-3-Opus次之;所有模型均出现信任破坏性和治疗后行为不当错误。
  • GPT-4可读性指标均优于Claude-3-Opus,FRE中位数69.01(IQR 64.88-73.14)vs 59.74(IQR 55.47-64.01)。

核心结论

  • GPT-4与Claude-3-Opus在生成简化介入放射学报告方面表现更优,但所有模型均存在错误(包括信任破坏性错误),凸显临床实施前需进一步改进和验证。

研究设计与样本

  • 比较分析研究,纳入109份介入放射学报告,评估专有模型(GPT-4、GPT-3.5 Turbo、Claude-3-Opus、Gemini Ultra)和开源模型(Mistral-7b、Mistral-8×7b)的简化性能。

诊断标准与关键数值

  • 定性评估采用五点Likert量表,涵盖准确性、完整性、清晰度、临床相关性、自然度和错误率(包括信任破坏性和治疗后行为不当错误);定量可读性采用FRE、FKGL、SMOG指数和DCRS;统计采用配对t检验和Bonferroni校正P值。

临床应用与阅片要点

  • 研究提供多种大语言模型重写介入放射学报告的性能数据,可帮助选择最适合以患者为中心的临床应用的模型。

局限性

  • 所有模型均表现出一定程度的信任破坏性和治疗后行为不当错误。
  • 所有模型(包括出现信任破坏性错误者)存在错误,凸显临床实施前需进一步改进和验证。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: To quantitatively and qualitatively evaluate and compare the performance of leading large language models (LLMs), including proprietary models (GPT-4, GPT-3.5 Turbo, Claude-3-Opus, and Gemini Ultra) and open-source models (Mistral-7b and Mistral-8×7b), in simplifying 109 interventional radiology reports.

METHODS: Qualitative performance was assessed using a five-point Likert scale for accuracy, completeness, clarity, clinical relevance, naturalness, and error rates, including trust-breaking and post-therapy misconduct errors. Quantitative readability was assessed using Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), SMOG Index, and Dale-Chall Readability Score (DCRS). Paired t-tests and Bonferroni-corrected p-values were used for statistical analysis.

RESULTS: Qualitative evaluation showed no significant differences between GPT-4 and Claude-3-Opus for any metrics evaluated (all Bonferroni-corrected p-values: p = 1), while they outperformed other assessed models across five qualitative metrics (p < 0.001). GPT-4 had the fewest content and trust-breaking errors, with Claude-3-Opus second. However, all models exhibited some level of trust-breaking and post-therapy misconduct errors, with GPT-4-Turbo and GPT-3.5-Turbo with few-shot prompting showing the lowest error rates, and Mistral-7B and Mistral-8×7B showing the highest. Quantitatively, GPT-4 surpassed Claude-3-Opus in all readability metrics (all p < 0.001), with a median FRE score of 69.01 (IQR: 64.88-73.14) versus 59.74 (IQR: 55.47-64.01) for Claude-3-Opus. GPT-4 also outperformed GPT-3.5-Turbo and Gemini Ultra (both p < 0.001). Inter-rater reliability was strong (κ = 0.77-0.84).

CONCLUSIONS: GPT-4 and Claude-3-Opus demonstrated superior performance in generating simplified IR reports, but the presence of errors across all models, including trust-breaking errors, highlights the need for further refinement and validation before clinical implementation.

CLINICAL RELEVANCE/APPLICATIONS: With the increasing complexity of interventional radiology (IR) procedures and the growing availability of electronic health records, simplifying IR reports is critical to improving patient understanding and clinical decision-making. This study provides insights into the performance of various LLMs in rewriting IR reports, which can help in selecting the most suitable model for clinical patient-centered applications.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Can E, Uller W, Vogt K, et al. Large Language Models for Simplified Interventional Radiology Reports: A Comparative Analysis. Acad Radiol. 2024. PMID: 39353826. doi:10.1016/j.acra.2024.09.041

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。