文献速递技术与安全神经头颈

多模态大语言模型在日本放射科医师资格考试(2021-2023年)中的表现

Performance of Multimodal Large Language Models in Japanese Diagnostic Radiology Board Examinations (2021-2023).
PubMed 文献 · Acad Radiol · 2024-11-08

本研究用2021至2023年日本放射科医师资格考试题目,在有图像和无图像两种条件下测试GPT-4o、Claude 3 Opus、GPT-4 Vision、Gemini Flash 1.5和Gemini Pro 1.5五个多模态大语言模型。各模型平均准确率为30.21%至45.00%,GPT-4o最高(45.00%),Claude 3 Opus无图像时最高(45.83%);加入图像未显著提高任何模型的准确率,且使一个模型准确率显著下降;所有模型均未达到60%的及格线。

要点

  • 五个多模态大语言模型在日本放射科医师资格考试中平均准确率为30.21%至45.00%,均未达到60%及格线。
  • 加入图像未显著提高任何模型的准确率,且使一个模型准确率显著下降。
  • 各模型在放射学亚专科间表现差异较大,GPT-4o在「其他」类别最高(65.63%),Claude 3 Opus在神经放射学与头颈部影像最高(55.56%)。

核心结论

  • 多模态大语言模型在日本放射科医师资格考试中表现出一定范围的准确率,GPT-4o和Claude 3 Opus总体表现最高,但加入图像未显著提高任何模型准确率;其评估放射医学图像的能力目前有限,常规使用前似乎需要进一步开发。

研究设计与样本

  • 用2021至2023年日本放射科医师资格考试题目,在有图像和无图像两种条件下测试GPT-4o、Claude 3 Opus、GPT-4 Vision、Gemini Flash 1.5和Gemini Pro 1.5五个多模态大语言模型,计算各模型总体及亚专科准确率。

诊断标准与关键数值

  • 各模型平均准确率为30.21%至45.00%,GPT-4o最高(45.00%),Claude 3 Opus无图像时最高(45.83%);及格阈值为60%,无模型超过;GPT-4o在「其他」类别准确率65.63%,Claude 3 Opus在神经放射学与头颈部影像准确率55.56%。

局限性

  • 加入图像未显著提高任何模型的准确率,且使一个模型准确率显著下降。
  • 所有模型均未超过60%的及格阈值。
  • 模型评估放射医学图像的能力目前有限,常规使用前似乎需要进一步开发。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: To evaluate the performance of various multimodal large language models (LLMs) in the Japanese Diagnostic Radiology Board Examinations (JDRBE) both with and without images.

MATERIALS AND METHODS: Five multimodal LLMs-GPT-4o, Claude 3 Opus, GPT-4 Vision, Gemini Flash 1.5, and Gemini Pro 1.5-were tested using questions from the JDRBE from 2021 to 2023. The models' performances were assessed in two conditions: with images and without images. Accuracy rates were calculated for each model, both overall and within specific subspecialties, including Abdominal and Pelvic Radiology, Musculoskeletal and Breast Imaging, Neuroradiology and Head and Neck Imaging, Nuclear Medicine, and Thoracic and Cardiac Radiology.

RESULTS: The average accuracy rates of the LLMs ranged from 30.21% to 45.00%, with GPT-4o achieving the highest (45.00%). Claude 3 Opus performed best without images (45.83%), while the addition of images did not significantly improve accuracy for any model. Performance varied across subspecialties, with GPT-4o excelling in "Other" (65.63%) and Claude 3 Opus in Neuroradiology and Head and Neck Imaging (55.56%). Importantly, none of the models surpassed the passing threshold of 60%.

CONCLUSION: Our findings demonstrate that multimodal LLMs exhibit a range of accuracy in JDRBE, with GPT-4o and Claude 3 Opus showing the highest overall performance. However, the addition of images did not significantly improve accuracy for any model.

SUMMARY: Multimodal LLMs are a very promising tool in the field of radiology. However, our study shows that while there are some promising results, their ability to evaluate radiological medical images is currently limited. Further development seems necessary before they can be used routinely.

KEY POINTS: Multimodal LLMs show varying accuracy (30.21-45.83%) on Japanese diagnostic radiology board examinations. Adding images did not significantly improve multimodal LLM performance, and significantly decreased accuracy for one model. Performances of multimodal LLMs varied considerably across radiology subspecialties.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Nakaura T, Yoshida N, Kobayashi N, et al. Performance of Multimodal Large Language Models in Japanese Diagnostic Radiology Board Examinations (2021-2023). Acad Radiol. 2024. PMID: 39521632. doi:10.1016/j.acra.2024.10.035

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。