文献速递技术与安全US

评估GPT-4o在欧洲放射学委员会官方考试中的表现:一项综合评估

Evaluating GPT-4o's Performance in the Official European Board of Radiology Exam: A Comprehensive Assessment.
PubMed 文献 · Acad Radiol · 2024-09-18

本研究将基于文本、图像或视频的多选题、自由文本报告及图像标注题以标准化提示上传至GPT-4o,并与实时参加欧洲放射学委员会(EBR)考试的放射科医生平均分比较。第一部分GPT-4o得分70.2%,高于放射科医生平均分58.4%和最高通过分60%;第二部分得分52.9%,低于放射科医生平均分66.1%和最低通过分55%。超声图像题准确率87.5-100%,视频题50.6%,最可能诊断题准确率100%,自由文本报告和直接解剖评估准确率分别为31%和28.6%。

要点

  • 第一部分GPT-4o得分70.2%,高于放射科医生平均分58.4%和最高通过分60%。
  • 第二部分GPT-4o得分52.9%,低于放射科医生平均分66.1%和最低通过分55%。
  • 最可能诊断题准确率100%,自由文本报告31%,直接解剖评估28.6%。

核心结论

  • GPT-4o在官方EBR考试中展现出值得注意的能力,本研究显示大语言模型即使在零样本提示下也有潜力辅助放射科医生从诊断到治疗或随访建议的病例评估与管理。

研究设计与样本

  • 本研究将基于文本、图像或视频的多选题、自由文本报告或图像标注题以标准化提示上传至GPT-4o,并将结果与实时参加考试的放射科医生平均分比较。

诊断标准与关键数值

  • 第一部分GPT-4o得分70.2%,放射科医生平均分58.4%,最高通过分60%。
  • 第二部分GPT-4o得分52.9%,放射科医生平均分66.1%,最低通过分55%。
  • 超声图像题准确率87.5-100%,视频题准确率50.6%。
  • 最可能诊断题准确率100%,自由文本报告准确率31%,直接解剖评估准确率28.6%。

临床应用与阅片要点

  • 本研究显示大语言模型在零样本提示下可能辅助放射科医生进行病例评估与管理。
  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: This study aims to evaluate the performance of generative pre-trained transformer (GPT)-4o in the complete official European Board of Radiology (EBR) exam, designed to assess radiology knowledge, skills, and competence.

MATERIALS AND METHODS: Questions based on text, image, or video and in the format of multiple choice, free-text reporting, or image annotation were uploaded into GPT-4o using standardized prompting. The results were compared to the average scores of radiologists taking the exam in real time.

RESULTS: In Part 1 (multiple response questions and short cases), GPT-4o outperformed both the radiologists' average scores and the maximum pass score (70.2% vs. 58.4% and 60%, respectively). In Part 2 (clinically oriented reasoning evaluation), the performance of GPT-4o was below both the radiologists' average scores and the minimum pass score (52.9% vs. 66.1% and 55%, respectively). The accuracy on questions involving ultrasound images was higher compared to other imaging modalities (accuracy rate, 87.5-100%). For video-based questions, the performance was 50.6%. The model achieved the highest accuracy on most likely diagnosis questions but showed lower accuracy in free-text reporting and direct anatomical assessment in images (100% vs. 31% and 28.6%, respectively).

CONCLUSION: The abilities of GPT-4o in the official EBR exam are particularly noteworthy. This study demonstrates the potential of large language models to assist radiologists in assessing and managing cases from diagnosis to treatment or follow-up recommendations, even with zero-shot prompting.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Beşler MS, Oleaga L, Junquero V, et al. Evaluating GPT-4o's Performance in the Official European Board of Radiology Exam: A Comprehensive Assessment. Acad Radiol. 2024. PMID: 39294055. doi:10.1016/j.acra.2024.09.005

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。