大语言模型用于简化介入放射学报告:比较分析
本研究对109份介入放射学报告,用专有模型(GPT-4、GPT-3.5 Turbo、Claude-3-Opus、Gemini Ultra)和开源模型(Mistral-7b、Mistral-8×7b)进行简化,从六个定性指标和四项可读性指标评估。GPT-4与Claude-3-Opus定性表现无显著差异且优于其他模型,GPT-4可读性优于Claude-3-Opus;所有模型均出现信任破坏性错误。
要点
- GPT-4与Claude-3-Opus在六项定性指标上无显著差异(Bonferroni校正后P=1),且优于其他评估模型。
- GPT-4内容错误和信任破坏性错误最少,Claude-3-Opus次之;所有模型均出现信任破坏性和治疗后行为不当错误。
- GPT-4可读性指标均优于Claude-3-Opus,FRE中位数69.01(IQR 64.88-73.14)vs 59.74(IQR 55.47-64.01)。
核心结论
- GPT-4与Claude-3-Opus在生成简化介入放射学报告方面表现更优,但所有模型均存在错误(包括信任破坏性错误),凸显临床实施前需进一步改进和验证。
研究设计与样本
- 比较分析研究,纳入109份介入放射学报告,评估专有模型(GPT-4、GPT-3.5 Turbo、Claude-3-Opus、Gemini Ultra)和开源模型(Mistral-7b、Mistral-8×7b)的简化性能。
诊断标准与关键数值
- 定性评估采用五点Likert量表,涵盖准确性、完整性、清晰度、临床相关性、自然度和错误率(包括信任破坏性和治疗后行为不当错误);定量可读性采用FRE、FKGL、SMOG指数和DCRS;统计采用配对t检验和Bonferroni校正P值。
临床应用与阅片要点
- 研究提供多种大语言模型重写介入放射学报告的性能数据,可帮助选择最适合以患者为中心的临床应用的模型。
局限性
- 所有模型均表现出一定程度的信任破坏性和治疗后行为不当错误。
- 所有模型(包括出现信任破坏性错误者)存在错误,凸显临床实施前需进一步改进和验证。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
PURPOSE: To quantitatively and qualitatively evaluate and compare the performance of leading large language models (LLMs), including proprietary models (GPT-4, GPT-3.5 Turbo, Claude-3-Opus, and Gemini Ultra) and open-source models (Mistral-7b and Mistral-8×7b), in simplifying 109 interventional radiology reports.
METHODS: Qualitative performance was assessed using a five-point Likert scale for accuracy, completeness, clarity, clinical relevance, naturalness, and error rates, including trust-breaking and post-therapy misconduct errors. Quantitative readability was assessed using Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), SMOG Index, and Dale-Chall Readability Score (DCRS). Paired t-tests and Bonferroni-corrected p-values were used for statistical analysis.
RESULTS: Qualitative evaluation showed no significant differences between GPT-4 and Claude-3-Opus for any metrics evaluated (all Bonferroni-corrected p-values: p = 1), while they outperformed other assessed models across five qualitative metrics (p < 0.001). GPT-4 had the fewest content and trust-breaking errors, with Claude-3-Opus second. However, all models exhibited some level of trust-breaking and post-therapy misconduct errors, with GPT-4-Turbo and GPT-3.5-Turbo with few-shot prompting showing the lowest error rates, and Mistral-7B and Mistral-8×7B showing the highest. Quantitatively, GPT-4 surpassed Claude-3-Opus in all readability metrics (all p < 0.001), with a median FRE score of 69.01 (IQR: 64.88-73.14) versus 59.74 (IQR: 55.47-64.01) for Claude-3-Opus. GPT-4 also outperformed GPT-3.5-Turbo and Gemini Ultra (both p < 0.001). Inter-rater reliability was strong (κ = 0.77-0.84).
CONCLUSIONS: GPT-4 and Claude-3-Opus demonstrated superior performance in generating simplified IR reports, but the presence of errors across all models, including trust-breaking errors, highlights the need for further refinement and validation before clinical implementation.
CLINICAL RELEVANCE/APPLICATIONS: With the increasing complexity of interventional radiology (IR) procedures and the growing availability of electronic health records, simplifying IR reports is critical to improving patient understanding and clinical decision-making. This study provides insights into the performance of various LLMs in rewriting IR reports, which can help in selecting the most suitable model for clinical patient-centered applications.
引用
- Can E, Uller W, Vogt K, et al. Large Language Models for Simplified Interventional Radiology Reports: A Comparative Analysis. Acad Radiol. 2024. PMID: 39353826. doi:10.1016/j.acra.2024.09.041
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。