文献速递技术与安全

具备视觉能力的大型语言模型在放射学委员会考试风格题目上的表现

Large Language Models with Vision on Diagnostic Radiology Board Exam Style Questions.
PubMed 文献 · Acad Radiol · 2024-12-04

研究回顾性选取280道放射学委员会考试风格题目,转换为多模态、纯图像和纯文本三种输入格式,评估GPT-4V、Gemini 1.5 Pro和Claude 3.5 Sonnet的表现。结果显示,GPT-4V和Gemini 1.5 Pro在不同输入格式下正确率无显著差异;Claude 3.5 Sonnet的纯图像输入正确率(48%)显著低于纯文本(63%)和多模态输入(66%),而纯文本与多模态之间无差异。结论认为具备视觉能力的大型语言模型无法有效利用图像提升答题表现,且Claude 3.5 Sonnet在纯文本和多模态格式下显著优于其他两个模型。

要点

  • GPT-4V和Gemini 1.5 Pro在纯文本、纯图像和多模态输入下正确率无显著差异。
  • Claude 3.5 Sonnet纯图像输入正确率48%,显著低于纯文本63%和多模态66%。
  • Claude 3.5 Sonnet在纯文本和多模态格式下显著优于GPT-4V和Gemini 1.5 Pro。

核心结论

  • 具备视觉能力的大型语言模型无法有效利用图像提升在放射学委员会考试风格题目上的表现;仅使用文本数据时,Claude 3.5 Sonnet优于GPT-4V和Gemini 1.5 Pro。

研究设计与样本

  • 回顾性研究,从AuntMinnie公共题库选取280道题目,转换为多模态、纯图像和纯文本三种输入格式,评估GPT-4V、Gemini 1.5 Pro和Claude 3.5 Sonnet三个模型,使用Cochran Q检验和成对McNemar检验比较。

诊断标准与关键数值

  • GPT-4V正确率:纯文本54%、纯图像52%、多模态57%(p = .31)。
  • Gemini 1.5 Pro正确率:纯文本53%、纯图像54%、多模态57%(p = .53)。
  • Claude 3.5 Sonnet正确率:纯图像48%显著低于纯文本63%(p < .001)和多模态66%(p < .001),纯文本与多模态间无差异(p = .29)。
  • Claude 3.5 Sonnet在纯文本和多模态格式下显著优于GPT-4V和Gemini 1.5 Pro(p < .01)。

临床应用与阅片要点

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: The expansion of large language models to process images offers new avenues for application in radiology. This study aims to assess the multimodal capabilities of contemporary large language models, which allow analysis of image inputs in addition to textual data, on radiology board-style examination questions with images.

MATERIALS AND METHODS: 280 questions were retrospectively selected from the AuntMinnie public test bank. The test questions were converted into three formats of prompts; (1) Multimodal, (2) Image-only, and (3) Text-only input. Three models, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet, were evaluated using these prompts. The Cochran Q test and pairwise McNemar test were used to compare performances between prompt formats and models.

RESULTS: No difference was found for the performance in terms of % correct answers between the text, image, and multimodal prompt formats for GPT-4V (54%, 52%, and 57%, respectively; p = .31) and Gemini 1.5 Pro (53%, 54%, and 57%, respectively; p = .53). For Claude 3.5 Sonnet, the image input (48%) significantly underperformed compared to the text input (63%, p < .001) and the multimodal input (66%, p < .001), but no difference was found between the text and multimodal inputs (p = .29). Claude significantly outperformed GPT and Gemini in the text and multimodal formats (p < .01).

CONCLUSION: Vision-capable large language models cannot effectively use images to increase performance on radiology board-style examination questions. When using textual data alone, Claude 3.5 Sonnet outperforms GPT-4V and Gemini 1.5 Pro, highlighting the advancements in the field and its potential for use in further research.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Sun SH, Chen K, Anavim S, et al. Large Language Models with Vision on Diagnostic Radiology Board Exam Style Questions. Acad Radiol. 2024. PMID: 39632215. doi:10.1016/j.acra.2024.11.028

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。