文献速递心脏核医学

评估人工智能在核心脏病学中的能力:大语言模型参加备考考试

Evaluating AI proficiency in nuclear cardiology: Large language models take on the board preparation exam.
PubMed 文献 · J Nucl Cardiol · 2024-11-29

本研究评估GPT-4、GPT-4 Turbo、GPT-4omni(GPT-4o)和Gemini四种大语言模型回答2023年美国核心脏病学会备考考试168道题(141道纯文本题和27道图像题)的表现,每个模型每部分重复30次。GPT-4o中位正确率63.1%(95%CI 62.5%-64.3%),显著优于其他模型,在纯文本题上表现优异,但图像题解读能力仍需改进。

要点

  • GPT-4o中位正确率63.1%(95%CI 62.5%-64.3%),在四种模型中表现最优。
  • GPT-4o在纯文本题上显著优于GPT-4、Gemini和GPT-4 Turbo(P值分别为<.001、<.001和.001)。
  • Gemini在图像题上表现较差(与其他模型比较P均<.001)。

核心结论

  • GPT-4o在四种大语言模型中表现最优,其得分可能处于通过类似CBNC考试所需范围的边缘或略低于该范围。
  • 尽管医学图像解读能力仍需改进,GPT-4o显示出辅助医生回答文本型临床问题的潜力。

研究设计与样本

  • 研究使用168道题(141道纯文本题和27道图像题),按CBNC考试分为四个部分。
  • 四种大语言模型(GPT-4、GPT-4 Turbo、GPT-4o和Gemini)接受相同的标准化提示词,每个部分重复30次以考虑随机性。
  • 除GPT-4o外,所有模型在六周内进行性能评估;使用McNemar检验比较正确回答比例。

主要发现

  • GPT-4、Gemini、GPT-4 Turbo和GPT-4o的中位正确率分别为56.8%(95%CI 55.4%-58.0%)、40.5%(95%CI 39.9%-42.9%)、60.7%(95%CI 59.5%-61.3%)和63.1%(95%CI 62.5%-64.3%)。
  • GPT-4o显著优于其他模型:与GPT-4 Turbo比较P=.007,与GPT-4和Gemini比较P<.001。
  • GPT-4o在纯文本题上显著优于GPT-4、Gemini和GPT-4 Turbo,P值分别为<.001、<.001和.001。
  • Gemini在图像题上表现更差,与其他所有模型比较P<.001。

关键数值与推荐

  • GPT-4中位正确率56.8%(95%CI 55.4%-58.0%)。
  • Gemini中位正确率40.5%(95%CI 39.9%-42.9%)。
  • GPT-4 Turbo中位正确率60.7%(95%CI 59.5%-61.3%)。
  • GPT-4o中位正确率63.1%(95%CI 62.5%-64.3%)。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Previous studies evaluated the ability of large language models (LLMs) in medical disciplines; however, few have focused on image analysis, and none specifically on cardiovascular imaging or nuclear cardiology. This study assesses four LLMs-GPT-4, GPT-4 Turbo, GPT-4omni (GPT-4o) (Open AI), and Gemini (Google Inc.)-in responding to questions from the 2023 American Society of Nuclear Cardiology Board Preparation Exam, reflecting the scope of the Certification Board of Nuclear Cardiology (CBNC) examination. METHODS: We used 168 questions: 141 text-only and 27 image-based, categorized into four sections mirroring the CBNC exam. Each LLM was presented with the same standardized prompt and applied to each section 30 times to account for stochasticity. Performance over six weeks was assessed for all models except GPT-4o. McNemar's test compared correct response proportions. RESULTS: GPT-4, Gemini, GPT-4 Turbo, and GPT-4o correctly answered median percentages of 56.8% (95% confidence interval 55.4% - 58.0%), 40.5% (39.9% - 42.9%), 60.7% (59.5% - 61.3%), and 63.1% (62.5%-64.3%) of questions, respectively. GPT-4o significantly outperformed other models (P = .007 vs GPT-4 Turbo, P < .001 vs GPT-4 and Gemini). GPT-4o excelled on text-only questions compared to GPT-4, Gemini, and GPT-4 Turbo (P < .001, P < .001, and P = .001), while Gemini performed worse on image-based questions (P < .001 for all). CONCLUSION: GPT-4o demonstrated superior performance among the four LLMs, achieving scores likely within or just outside the range required to pass a test akin to the CBNC examination. Although improvements in medical image interpretation are needed, GPT-4o shows potential to support physicians in answering text-based clinical questions.

来源:J Nucl Cardiol,查看原文。版权归原作者所有。

引用

  1. Builoff V, Shanbhag A, Miller RJ, et al. Evaluating AI proficiency in nuclear cardiology: Large language models take on the board preparation exam. J Nucl Cardiol. 2024. PMID: 39617127. doi:10.1016/j.nuclcard.2024.102089

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。