文献速递技术与安全

评估GPT大型语言模型在RSNA 2023每日病例问题上的表现

Evaluation of GPT Large Language Model Performance on RSNA 2023 Case of the Day Questions.
PubMed 文献 · Radiology · 2024-01-01

本研究评估GPT-4V在RSNA 2023每日病例问题上的准确性,并与5名放射科医生和3名住院医师比较。GPT-4V准确率为43%(31/72;95%CI:32,55)。放射科医生和住院医师在依赖影像和不依赖影像的病例中均未显著优于GPT-4V,且GPT-4V辅助未提高读片者平均准确率。

要点

  • GPT-4V在影像病例问题上的准确率为43%,与放射科医生和住院医师无显著差异。
  • GPT-4V仅文本输入准确率50%,仅图像输入准确率38%,提示其依赖文本上下文。
  • 提供GPT-4V输出未改善放射科医生和住院医师的平均准确率。

核心结论

  • 放射科医生和住院医师未显著优于GPT-4V,GPT-4V辅助未帮助人类读片者,GPT-4V依赖文本上下文进行输出。

研究设计与样本

  • 观察性研究,纳入RSNA 2023年会72个每日病例问题,由5名放射科医生和3名住院医师在开卷条件下作答,并与GPT-4V在仅图像、仅文本及文本加图像输入下的答案比较,使用混合效应线性模型和McNemar检验,P<.05为显著。

诊断标准与关键数值

  • GPT-4V总体准确率43%(31/72;95%CI:32,55);仅文本输入准确率50%(35/70;95%CI:39,61);仅图像输入准确率38%(26/69;95%CI:27,49)。

临床应用与阅片要点

  • 原文未据此给出推荐。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

Background GPT-4V (GPT-4 with vision, ChatGPT; OpenAI) has shown impressive performance in several medical assessments. However, few studies have assessed its performance in interpreting radiologic images. Purpose To assess and compare the accuracy of GPT-4V in assessing radiologic cases with both images and textual context to that of radiologists and residents, to assess if GPT-4V assistance improves human accuracy, and to assess and compare the accuracy of GPT-4V with that of image-only or text-only inputs. Materials and Methods Seventy-two Case of the Day questions at the RSNA 2023 Annual Meeting were curated in this observer study. Answers from GPT-4V were obtained between November 26 and December 10, 2023, with the following inputs for each question: image only, text only, and both text and images. Five radiologists and three residents also answered the questions in an "open book" setting. For the artificial intelligence (AI)-assisted portion, the radiologists and residents were provided with the outputs of GPT-4V. The accuracy of radiologists and residents, both with and without AI assistance, was analyzed using a mixed-effects linear model. The accuracies of GPT-4V with different input combinations were compared by using the McNemar test. P < .05 was considered to indicate a significant difference. Results The accuracy of GPT-4V was 43% (31 of 72; 95% CI: 32, 55). Radiologists and residents did not significantly outperform GPT-4V in either imaging-dependent (59% and 56% vs 39%; P = .31 and .52, respectively) or imaging-independent (76% and 63% vs 70%; both P = .99) cases. With access to GPT-4V responses, there was no evidence of improvement in the average accuracy of the readers. The accuracy obtained by GPT-4V with text-only and image-only inputs was 50% (35 of 70; 95% CI: 39, 61) and 38% (26 of 69; 95% CI: 27, 49), respectively. Conclusion The radiologists and residents did not significantly outperform GPT-4V. Assistance from GPT-4V did not help human raters. GPT-4V relied on the textual context for its outputs. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Katz in this issue.

来源:Radiology,查看原文。版权归原作者所有。

引用

  1. Mukherjee P, Hou B, Suri A, et al. Evaluation of GPT Large Language Model Performance on RSNA 2023 Case of the Day Questions. Radiology. 2024. PMID: 39352277. doi:10.1148/radiol.240609

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。