文献速递神经MRCT

多模态大语言模型神经放射学逐题评估:代际性能与执行变异性

Item-Level Evaluation of Multimodal Large Language Models in Neuroradiology: Generational Performance and Execution Variability.
PubMed 文献 · AJNR Am J Neuroradiol · 2026-06-13

本研究以106道源自Radiopaedia的神经放射学图像多选题,比较神经放射学专家、住院医师及四种视觉大语言模型(GPT-4、GPT-5、Gemini 1.5、Gemini 2.5)的逐题表现。结果显示第二代模型虽有代际提升并接近或超过住院医师,但与专家参考仍有显著差距,且平均准确率改善并不等同于执行一致性的改善。

要点

  • 大语言模型在神经放射学任务中有代际提升,但仍未收敛至专家水平。
  • 模型平均准确率的相对稳定不代表单次决策层面可靠,需关注执行变异性。
  • 逐题、配对、以人为参照的评估能更真实地区分表面性能增益与专家一致性。

核心结论

  • 多模态大语言模型在神经放射学任务中虽有显著代际提升,但并未构成向专家参考性能的收敛。
  • 逐题、配对、以人为参照的评估为解读基准性能提供了关键背景,有助于区分表面性能增益与专家参照的真正一致性。
  • 聚合准确率的相对稳定并不必然意味着个体决策层面的可靠性,需同时评估性能幅度与执行层面一致性。

研究设计与样本

  • 研究设计为基于106道源自Radiopaedia的图像型神经放射学多选题的逐题评估。
  • 比较对象包括神经放射学专家、放射科住院医师及四种视觉大语言模型(GPT-4、GPT-5、Gemini 1.5、Gemini 2.5)。
  • 分析在题目层面进行,保留各组间逐题配对;平均准确率差异采用非参数bootstrap 95%置信区间估计。
  • 统计推断采用配对置换检验,限定于预先设定的对比,主要比较采用错误发现率校正。
  • 另以Radiopaedia社区准确率作为社区层面参照,对模型表现进行情境化对照。

诊断标准与关键数值

  • 神经放射学专家平均逐题准确率最高,为0.915(95%置信区间0.877–0.953)。
  • GPT-5与Gemini 2.5相较于专家参照,平均每题准确率差异分别为-0.236和-0.217。
  • 第二代模型在部分预设对比中接近或超过住院医师层面表现。

临床应用与阅片要点

  • 引入大语言模型辅助神经放射学阅片时,不应以聚合准确率作为唯一可靠性指标。
  • 在临床决策场景中,模型单次执行结果的不一致性可能造成错误判读风险,需谨慎使用。
  • 以专家参照为基准的逐题配对评估比社区平均准确率更能反映模型的实际专家对齐程度。

局限性

  • 题目来源于Radiopaedia,社区层面准确率不代表专家标准,仅作为聚合学习者层面的参照。
  • 模型执行层面的一致性分析虽被提出,但平均准确率的改善并未被证实伴随执行一致性的改善。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

BACKGROUND: Multimodal large language models have demonstrated consistent generational improvements on medical benchmark tasks, including radiology applications. However, whether these gains represent meaningful convergence toward the expert reference performance in subspecialty imaging domains such as neuroradiology remains uncertain, particularly when evaluated using item-level, human-referenced designs.

METHODS: We compared expert neuroradiologists, radiology residents, and four vision-capable large language models (GPT-4, GPT-5, Gemini 1.5, Gemini 2.5) using 106 image-based neuroradiology multiple-choice questions derived from Radiopaedia. Analyses were conducted at the question (item) level, preserving within-question pairing across groups. Mean accuracy differences were estimated using non-parametric bootstrap 95% confidence intervals, and statistical inference was performed using paired permutation tests restricted to pre-specified contrasts with false discovery rate correction for primary comparisons. Repeated model executions were analyzed to characterize execution-level variability. Performance was further contextualized against Radiopaedia community accuracy as a community-level reference.

RESULTS: Expert neuroradiologists achieved the highest mean item-level accuracy (0.915; 95% confidence interval, 0.877-0.953). Second-generation models demonstrated improved mean accuracy relative to earlier versions and approximated or exceeded resident-level performance in selected comparisons. However, a substantial gap relative to the expert reference persisted. GPT-5 and Gemini 2.5 underperformed the expert reference by mean per-item differences of -0.236 and -0.217, respectively. When contextualized against the community-level reference, advanced models aligned more closely with aggregate learner performance than with the expert reference accuracy. Improvements in mean accuracy were not uniformly accompanied by improved execution consistency.

CONCLUSIONS: Although multimodal large language models show meaningful generational gains on neuroradiology tasks, these improvements do not constitute convergence toward the expert reference performance. Item-level, paired, human-referenced evaluation provides critical context for interpreting benchmark performance and helps distinguish apparent performance gains from true alignment with the expert reference. Importantly, relative stability in aggregate accuracy does not necessarily imply reliability at the level of individual decisions, underscoring the need to assess both performance magnitude and execution-level consistency.

来源:AJNR Am J Neuroradiol,查看原文。版权归原作者所有。

引用

  1. Ojeda-Esparza, Yun, Meinzer, et al. Item-Level Evaluation of Multimodal Large Language Models in Neuroradiology: Generational Performance and Execution Variability. AJNR Am J Neuroradiol. 2026. PMID: 42288333. doi:10.3174/ajnr.A9477

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。