文献速递技术与安全

前沿大语言模型在ACR住院医师培训考试中的表现比较:2025年更新

Performance Comparison of Cutting-Edge Large Language Models on the ACR In-Training Examination: An Update for 2025.
PubMed 文献 · Acad Radiol · 2025-09-24

研究用106道公开的ACR放射诊断学住院医师培训考试题目评估7个大语言模型,GPT-o1未校正准确率最高(71.7%),GPT-4o在文本题上显著优于图像题;63道图像题改写后准确率49.2% vs 匹配原题56.1%,差异无统计学显著,提示数据污染可能作用有限。

要点

  • 106道DXIT题中GPT-o1未校正准确率最高(71.7%),GPT-4o(69.8%)与GPT-o3(68.9%)紧随其后。
  • GPT-4o在文本题上的表现显著高于图像题。
  • 改写图像题未校正准确率49.2% vs 匹配原题56.1%,两者无显著差异,提示数据污染可能作用有限。

核心结论

  • 现代大语言模型(尤其OpenAI系列)在放射科考试风格评估中表现出色且较前提升,改写题与原题表现相当提示数据污染可能作用有限。
  • 随着大语言模型进步,其通过个性化反馈和考试风格题目复习支持放射科住院医师学习具有较强潜力。

研究设计与样本

  • 研究评估7个大语言模型:GPT-4、GPT-4o、GPT-o1、GPT-o3、Claude 3.5 Sonnet、Gemini 1.5 Pro和Grok 2.0。
  • 使用106道公开可得的ACR放射诊断学住院医师培训考试(DXIT)题目,以标准化指令集提示模型模拟放射科住院医师答题。
  • 记录每个模型选择的答案、理由和置信度评分,计算未校正准确率和逻辑校正准确率。
  • 亚组分析比较文本题与图像题的表现;另将63道图像题改写以测试新推理,同时保留原始诊断图像以评估潜在训练数据污染的影响。

诊断标准与关键数值

  • 未校正准确率(基于正确选择答案):GPT-o1 71.7%、GPT-4o 69.8%、GPT-o3 68.9%、GPT-4 59.4%、Grok 2.0 52.8%、Claude 3.5 Sonnet 34.9%。
  • 逻辑校正准确率(基于临床推理路径):GPT-o1 60.4%、GPT-4o 59.4%、GPT-o3 59.4%、Grok 2.0 34.0%、Claude 3.5 Sonnet 30.2%。
  • 改写DXIT题未校正准确率49.2% vs 匹配原题56.1%,逻辑校正准确率40.0% vs 匹配原题44.4%,原题与改写题之间无显著差异。

临床应用与阅片要点

  • 评估大语言模型答题能力时需区分未校正准确率与逻辑校正准确率,后者反映临床推理路径的正确性。
  • GPT-4o在文本题上的表现显著高于图像题,提示多模态推理能力仍有提升空间。
  • 改写图像题与原题表现无显著差异,提示数据污染可能对模型在DXIT题目上的表现作用有限。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: This study represents a continuation of prior work by Payne et al. evaluating large language model (LLM) performance on radiology board-style assessments, specifically the ACR diagnostic radiology in-training examination (DXIT). Building upon earlier findings with GPT-4, we assess the performance of newer, cutting-edge models, such as GPT-4o, GPT-o1, GPT-o3, Claude, Gemini, and Grok on standardized DXIT questions. In addition to overall performance, we compare model accuracy on text-based versus image-based questions to assess multi-modal reasoning capabilities. As a secondary aim, we investigate the potential impact of data contamination by comparing model performance on original versus revised image-based questions.

METHODS: Seven LLMs - GPT-4, GPT-4o, GPT-o1, GPT-o3, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Grok 2.0-were evaluated using 106 publicly available DXIT questions. Each model was prompted using a standardized instruction set to simulate a radiology resident answering board-style questions. For each question, the model's selected answer, rationale, and confidence score were recorded. Unadjusted accuracy (based on correct answer selection) and logic-adjusted accuracy (based on clinical reasoning pathways) were calculated. Subgroup analysis compared model performance on text-based versus image-based questions. Additionally, 63 image-based questions were revised to test novel reasoning while preserving the original diagnostic image to assess the impact of potential training data contamination.

RESULTS: Across 106 DXIT questions, GPT-o1 demonstrated the highest unadjusted accuracy (71.7%), followed closely by GPT-4o (69.8%) and GPT-o3 (68.9%). GPT-4 (59.4%) and Grok 2.0 exhibited similar scores (59.4% and 52.8%). Claude 3.5 Sonnet had the lowest unadjusted accuracies (34.9%). Similar trends were observed for logic-adjusted accuracy, with GPT-o1 (60.4%), GPT-4o (59.4%), and GPT-o3 (59.4%) again outperforming other models, while Grok 2.0 and Claude 3.5 Sonnet lagged behind (34.0% and 30.2%, respectively). GPT-4o's performance was significantly higher on text-based questions compared to image-based ones. Unadjusted accuracy for the revised DXIT questions was 49.2%, compared to 56.1% on matched original DXIT questions. Logic-adjusted accuracy for the revised DXIT questions was 40.0% compared to 44.4% on matched original DXIT questions. No significant difference in performance was observed between original and revised questions.

CONCLUSION: Modern LLMs, especially those from OpenAI, demonstrate strong and improved performance on board-style radiology assessments. Comparable performance on revised prompts suggests that data contamination may have played a limited role. As LLMs improve, they hold strong potential to support radiology resident learning through personalized feedback and board-style question review.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Young A, Paloka R, Islam A, et al. Performance Comparison of Cutting-Edge Large Language Models on the ACR In-Training Examination: An Update for 2025. Acad Radiol. 2025. PMID: 40998658. doi:10.1016/j.acra.2025.09.008

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。