评估推理型大语言模型在日本放射科委员会考试题目上的表现
研究将2022-2023年日本放射学会官方委员会考试210道题输入四种大语言模型,o1-preview准确率最高(85.7%),显著优于GPT-4o(73.3%,P<.001),o1-mini(69.5%)也显著优于GPT-4o-mini(46.7%,P<.001),但推理模型成本更高、处理时间更长。
要点
- o1-preview在全部放射亚专科中准确率均最高(85.7%)。
- 推理模型准确率显著优于基座模型,但成本和处理时间大幅增加。
- 该研究仅评估答题准确率,未涉及临床阅片场景。
核心结论
- 推理型大语言模型在回答放射科委员会考试题目上表现明显优于其基座模型,但成本大幅增加且处理时间延长。
研究设计与样本
- 研究论文;共210道来自2022-2023年日本放射学会官方委员会考试的题目,输入四种大语言模型(o1-preview、o1-mini、GPT-4o、GPT-4o-mini),按六个预设放射亚专科计算答题正确百分比,记录总成本和每模型处理时间,采用McNemar检验比较配对模型准确率差异的统计学显著性。
诊断标准与关键数值
- o1-preview准确率85.7%,显著优于GPT-4o的73.3%(P<.001)。
- o1-mini准确率69.5%,显著优于GPT-4o-mini的46.7%(P<.001)。
- o1-preview在所有放射亚专科中排名最高。
临床应用与阅片要点
- 推理模型(o1-preview)成本为17.10美元,o1-mini为2.58美元,基座模型GPT-4o为0.496美元、GPT-4o-mini为0.04美元。
- 推理模型处理时间分别约为对应基座模型的3.7倍和1.2倍。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: This study evaluates the performance, cost, and processing time of OpenAI's reasoning large language models (LLMs) (o1-preview, o1-mini) and their base models (GPT-4o, GPT-4o-mini) on Japanese radiology board examination questions.
MATERIALS AND METHODS: A total of 210 questions from the 2022-2023 official board examinations of the Japan Radiological Society were presented to each of the four LLMs. Performance was evaluated by calculating the percentage of correctly answered questions within six predefined radiology subspecialties. The total cost and processing time for each model were also recorded. The McNemar test was used to assess the statistical significance of differences in accuracy between paired model responses.
RESULTS: The o1-preview achieved the highest accuracy (85.7%), significantly outperforming GPT-4o (73.3%, P<.001). Similarly, o1-mini (69.5%) performed significantly better than GPT-4o-mini (46.7%, P<.001). Across all radiology subspecialties, o1-preview consistently ranked highest. However, reasoning models incurred substantially higher costs (o1-preview: $17.10, o1-mini: $2.58) compared to their base counterparts (GPT-4o: $0.496, GPT-4o-mini: $0.04), and their processing times were approximately 3.7 and 1.2 times longer, respectively.
CONCLUSION: Reasoning LLMs demonstrated markedly superior performance in answering radiology board exam questions compared to their base models, albeit at a substantially higher cost and increased processing time.
引用
- Nakaura T, Takamure H, Kobayashi N, et al. Evaluating the Performance of Reasoning Large Language Models on Japanese Radiology Board Examination Questions. Acad Radiol. 2025. PMID: 40383659. doi:10.1016/j.acra.2025.04.060
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。