改进放射学诊断:利用测验病例探究GPT-3.5 Turbo与GPT-4的效用与局限性
本研究从美国神经放射学杂志网站「每周病例」系列选取2021年10月至2023年12月的115个病例,将描述与图注部分提供给GPT-3.5 Turbo和GPT-4完成直接诊断与病例报告检索两项任务。GPT-3.5 Turbo直接诊断正确率为26%(30/115),GPT-4为41%(47/115);病例报告检索正确率分别为10%(11/115)和7%(8/115)。在GPT-3.5 Turbo中联合病例报告检索可提升整体正确应答(P=0.023),而GPT-4未见显著提升(P=0.248)。
要点
- GPT-4直接诊断正确率为41%(47/115),GPT-3.5 Turbo为26%(30/115)。
- 病例报告检索任务正确率低:GPT-3.5 Turbo 10%(11/115),GPT-4 7%(8/115)。
- 联合病例报告检索仅改善GPT-3.5 Turbo表现(P=0.023),GPT-4无显著改善(P=0.248)。
核心结论
- 联合病例报告检索对GPT-3.5 Turbo的提升尤为明显,提示其可能作为GPTs直接诊断不可得时的替代诊断途径;但GPT模型在直接诊断与病例报告检索任务中的整体表现仍不理想,用户应意识到其局限性。
研究设计与样本
- 研究选取2021年10月至2023年12月美国神经放射学杂志网站「每周病例」系列的115个病例,将描述与图注部分提供给GPT-3.5 Turbo和GPT-4完成直接诊断与病例报告检索两项任务;直接诊断任务要求模型提供3个鉴别诊断,与诊断部分匹配即视为正确;病例报告检索任务要求模型每例生成2个关键词构建PubMed检索查询,提取至多3篇相关报告,提取到包含诊断部分所述疾病名称的报告即视为正确;采用McNemar检验评估在直接诊断基础上增加病例报告检索是否提高整体正确率。
诊断标准与关键数值
- 直接诊断任务正确率:GPT-3.5 Turbo 26%(30/115),GPT-4 41%(47/115);病例报告检索任务正确率:GPT-3.5 Turbo 10%(11/115),GPT-4 7%(8/115);联合病例报告检索后总正确应答:GPT-3.5 Turbo 32%(37/115,3例重叠),GPT-4 43%(50/115,5例重叠)。
临床应用与阅片要点
- 作者认为,联合病例报告检索对GPT-3.5 Turbo的提升尤为明显,提示其可能作为GPTs直接诊断不可得时的替代诊断途径;同时指出GPT模型在直接诊断与病例报告检索任务中的整体表现仍不理想,用户应意识到其局限性。
局限性
- GPT模型在直接诊断与病例报告检索任务中的整体表现仍不理想。
- 用户应意识到GPT模型的局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
BACKGROUND AND PURPOSE: The rise of large language models such as generative pretrained transformers (GPTs) has sparked considerable interest in radiology, especially in interpreting radiologic reports and image findings. While existing research has focused on GPTs estimating diagnoses from radiologic descriptions, exploring alternative diagnostic information sources is also crucial. This study introduces the use of GPTs (GPT-3.5 Turbo and GPT-4) for information retrieval and summarization, searching relevant case reports via PubMed, and investigates their potential to aid diagnosis.
MATERIALS AND METHODS: From October 2021 to December 2023, we selected 115 cases from the "Case of the Week" series on the American Journal of Neuroradiology website. Their Description and Legend sections were presented to the GPTs for the 2 tasks. For the Direct Diagnosis task, the models provided 3 differential diagnoses that were considered correct if they matched the diagnosis in the diagnosis section. For the Case Report Search task, the models generated 2 keywords per case, creating PubMed search queries to extract up to 3 relevant reports. A response was considered correct if reports containing the disease name stated in the diagnosis section were extracted. The McNemar test was used to evaluate whether adding a Case Report Search to Direct Diagnosis improved overall accuracy.
RESULTS: In the Direct Diagnosis task, GPT-3.5 Turbo achieved a correct response rate of 26% (30/115 cases), whereas GPT-4 achieved 41% (47/115). For the Case Report Search task, GPT-3.5 Turbo scored 10% (11/115), and GPT-4 scored 7% (8/115). Correct responses totaled 32% (37/115) with 3 overlapping cases for GPT-3.5 Turbo, whereas GPT-4 had 43% (50/115) of correct responses with 5 overlapping cases. Adding Case Report Search improved GPT-3.5 Turbo's performance (P = .023) but not that of GPT-4 (P = .248).
CONCLUSIONS: The effectiveness of adding Case Report Search to GPT-3.5 Turbo was particularly pronounced, suggesting its potential as an alternative diagnostic approach to GPTs, particularly in scenarios where direct diagnoses from GPTs are not obtainable. Nevertheless, the overall performance of GPT models in both direct diagnosis and case report retrieval tasks remains not optimal, and users should be aware of their limitations.
引用
- Kikuchi T, Nakao T, Nakamura Y, et al. Toward Improved Radiologic Diagnostics: Investigating the Utility and Limitations of GPT-3.5 Turbo and GPT-4 with Quiz Cases. AJNR Am J Neuroradiol. 2024. PMID: 38719605. doi:10.3174/ajnr.A8332
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。