文献速递技术与安全CTMR

前沿大语言模型在FRCR Part 1风格考试中的表现比较

Performance of Frontier Large Language Models on an FRCR Part 1-Style Examination: A Comparative Study of Physics and Anatomy Modules.
PubMed 文献 · Acad Radiol · 2026-08-31

本研究比较了三种前沿大语言模型在模拟FRCR Part 1考试中的表现。物理模块含40道判断题(200个评分点),解剖模块含100道图像题(0-2分制)。结果显示模型在文本型物理题上表现强劲,在图像解剖题上差异显著。

要点

  • 大语言模型在文本型物理题上表现优异,可用于放射科医师备考辅助。
  • 图像解剖题上模型间差异显著,阅片与精确定位仍是短板。
  • 可将此类模型用于教学场景,但不宜替代系统解剖训练。

核心结论

  • 大语言模型在文本型物理题上表现强劲,但在图像解剖题上表现差异较大,提示可用于教学辅助但存在解剖定位局限。

研究设计与样本

  • 研究类型为基于题库的横断面比较研究,使用定制的FRCR Part 1风格考试。
  • 物理模块含40道判断题共200个独立评分点,解剖模块含100道基于图像的题目按0-2分制评分。
  • 试题结构按照已发表的FRCR Part 1主题与成像方式分布设计,由两名已通过该考试的放射科医师审校。
  • 评估对象为GPT-5.4 Thinking、Gemini 3.1 Pro和Claude Opus 4.6三种大语言模型。
  • 物理模块比较采用Cochran Q检验与精确McNemar检验,解剖模块采用Friedman检验与Wilcoxon符号秩检验。

诊断标准与关键数值

  • 物理模块得分为Claude 195/200、GPT 189/200、Gemini 187/200,总体P=0.018。
  • 物理模块Holm校正后Claude优于Gemini(校正P=0.023),其余两两比较差异无统计学意义。
  • 解剖模块得分为Gemini 160/200、GPT 137/200、Claude 83/200,总体P<0.001。
  • 解剖模块所有两两比较经Holm校正后仍具有统计学意义(校正P≤0.038)。

临床应用与阅片要点

  • 大语言模型可作为放射科医师备考FRCR Part 1物理模块的辅助工具。
  • 在涉及图像解剖定位的训练中不宜依赖此类模型,需结合真实影像与解剖教学。

局限性

  • 研究使用自定义模拟试题而非官方FRCR Part 1考试真题,结论外推需谨慎。
  • 试题由两名放射科医师审校,但样本量和审校流程的稳健性有限。
  • 研究仅评估三种模型,未覆盖其他主流大语言模型版本。
  • 图像题评分为0-2分制,可能无法完全反映临床影像解读的细微差异。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: This study aims to compare the performance of three frontier large language models on an Fellowship of the Royal College of Radiologists (FRCR) Part 1-style examination comprising physics and image-based anatomy modules.

MATERIALS AND METHODS: GPT-5.4 Thinking, Gemini 3.1 Pro, and Claude Opus 4.6 were evaluated using a custom FRCR Part 1-style examination. The physics module included 40 true/false questions comprising 200 independently scored statements, while the anatomy module included 100 image-based questions scored on a 0-2 scale. Both modules were structured according to the published FRCR Part 1 topic and modality distributions and were reviewed by two radiologists who had previously passed the examination. Model performance was compared using Cochran's Q and exact McNemar tests for physics and Friedman and Wilcoxon signed-rank tests for anatomy.

RESULTS: Physics scores were 195/200 for Claude, 189/200 for GPT, and 187/200 for Gemini (overall P = 0.018). After Holm correction, Claude outperformed Gemini (adjusted P = 0.023), whereas the other pairwise differences were not significant. Anatomy scores were 160/200 for Gemini, 137/200 for GPT, and 83/200 for Claude (overall P < 0.001); all pairwise comparisons remained significant after Holm correction (adjusted P ≤ 0.038).

CONCLUSION: In this custom FRCR Part 1-style assessment, large language models performed strongly on text-based physics questions but showed more variable performance on image-based anatomy, supporting their potential educational role while highlighting persistent limitations in precise anatomical localization.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Güzel, Ünverdi. Performance of Frontier Large Language Models on an FRCR Part 1-Style Examination: A Comparative Study of Physics and Anatomy Modules. Acad Radiol. 2026. PMID: 42674933. doi:10.1016/j.acra.2026.08.077

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。