文献速递技术与安全乳腺介入

大型语言模型在放射科住院医师培训考试题目上的表现

Performance of Large Language Models on Radiology Residency In-Training Examination Questions.
PubMed 文献 · Acad Radiol · 2025-11-11

本研究分析2023至2025年间282道放射科住院医师考试多选题(191道纯文本、91道含图像),测试ChatGPT-4o、ChatGPT-5、Claude 4 Opus、Claude 4.1 Opus和Gemini 2.5 Pro五种大型语言模型,并与放射科住院医师表现对比。Gemini 2.5 Pro总体准确率最高(83.0%),其次为ChatGPT-5(82.3%),住院医师总体准确率为78.2%;含图像题中住院医师准确率最高(80.4%),未观察到幻觉。

要点

  • 含图像题中住院医师准确率最高(80.4%),ChatGPT-5次之(73.6%),模型在图像判读上仍逊于住院医师。
  • 纯文本题中Gemini 2.5 Pro准确率最高(88.0%),模型对文字类题目表现较好。
  • 各模型在乳腺影像亚专科准确率最低,在介入放射学与物理学准确率最高。

核心结论

  • 大型语言模型在放射科住院医师评估中总体表现良好,较新版本有可测量的提升,但在图像判读和某些亚专科上仍存在局限。
  • 作者认为大型语言模型应被视为放射科教育的辅助资源,并需谨慎验证和持续完善医学训练数据。

研究设计与样本

  • 研究类型为对放射科住院医师培训考试题目的模型性能评估,分析2023至2025年间本院放射科住院医师考试的多选题共282道(191道纯文本、91道含图像)。
  • 测试五种大型语言模型:ChatGPT-4o、ChatGPT-5、Claude 4 Opus、Claude 4.1 Opus和Gemini 2.5 Pro。
  • 同时分析放射科住院医师在同一套题目上的表现作为对比,并评估模型输出是否存在幻觉。
  • 准确率按总体、纯文本题和含图像题分别计算,比较采用Cochran Q检验和Bonferroni校正的McNemar检验。

诊断标准与关键数值

  • 总体准确率:Gemini 2.5 Pro最高(83.0%),其次为ChatGPT-5(82.3%),放射科住院医师为78.2%。
  • 纯文本题准确率:Gemini 2.5 Pro最高(88.0%)。
  • 含图像题准确率:放射科住院医师最高(80.4%),其次为ChatGPT-5(73.6%)。
  • ChatGPT-5准确率显著高于ChatGPT-4o(p = 0.021);Gemini 2.5 Pro准确率显著高于Claude 4 Opus(p = 0.026)。
  • 各亚专科中,介入放射学和物理学准确率最高,乳腺影像在所有模型中准确率最低。

临床应用与阅片要点

  • 作者认为大型语言模型应被视为放射科教育的辅助资源,需谨慎验证并持续完善医学训练数据。
  • 原文未据此给出针对日常阅片和报告流程的推荐。

局限性

  • 原文指出大型语言模型在图像判读和某些亚专科上仍存在局限。
  • 原文结论提出需对大型语言模型进行谨慎验证并持续完善医学训练数据。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: Large language models (LLMs) are increasingly investigated in radiology education. This study evaluated the performance of several advanced LLMs on radiology residency in-training examination questions, with a focus on whether recently released versions show improved accuracy compared with earlier models.

MATERIALS AND METHODS: We analyzed 282 multiple-choice questions (191 text-only, 91 image-based) from institutional radiology residency examinations conducted between 2023 and 2025. Five LLMs were tested: ChatGPT-4o, ChatGPT-5, Claude 4 Opus, Claude 4.1 Opus, and Gemini 2.5 Pro. Radiology resident performance on the same set of questions was also analyzed for comparison. Accuracy rates were calculated for overall, text-only, and image-based questions, and results were compared using Cochran's Q and Bonferroni-adjusted McNemar tests. Outputs were also assessed for hallucinations.

RESULTS: Gemini 2.5 Pro achieved the highest overall accuracy (83.0%), followed by ChatGPT-5 (82.3%). By comparison, radiology residents achieved an overall accuracy of 78.2%. ChatGPT-5 showed significantly higher accuracy compared with ChatGPT-4o (p = 0.021), and Gemini 2.5 Pro showed significantly higher accuracy compared with Claude 4 Opus (p = 0.026). For text-only questions, the highest accuracy was obtained with Gemini 2.5 Pro (88.0%). For image-based questions, radiology residents achieved the highest accuracy (80.4%), followed by ChatGPT-5 (73.6%). The highest accuracies by subspecialty were observed in interventional radiology and physics, whereas breast imaging yielded the lowest accuracy across the models. No instances of hallucination were observed.

CONCLUSION: LLMs demonstrated generally good performance on radiology residency assessments, with newer versions showing measurable improvements. However, limitations persist in image-based interpretation and certain subspecialties. LLMs should therefore be regarded as supportive resources in radiology education, with careful validation and continued refinement of medical training data.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Salbas A, Yogurtcu M. Performance of Large Language Models on Radiology Residency In-Training Examination Questions. Acad Radiol. 2025. PMID: 41224539. doi:10.1016/j.acra.2025.10.043

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。