多模态大模型放射影像解读的诊断准确率与资源消耗关系
本研究利用73例超声医学病例评估三种多模态推理模型在纯影像与影像加描述文本条件下、三种提示策略中的诊断表现及资源消耗。结果显示,o1在链式思考提示下多选题准确率为56.2%,超过人类基准55.9%;加入描述文本后o1最高准确率达71.2%。结论认为模型架构决定基线性能,提示优化主要影响效率。
要点
- o1在链式思考提示下多选准确率56.2%,略高于人类基准55.9%。
- 加入描述文本可明显提高所有模型准确率,o1最高达71.2%。
- o1输出token数与准确率呈负相关,提示准确不等于输出更长。
核心结论
- 模型架构从根本上决定基线性能,提示优化仅能在架构约束内影响效率。
研究设计与样本
- 研究纳入73例来自韩国超声医学学会网站的病例,并附有人类基准数据。
- 评估三种多模态推理模型在纯影像与影像加描述文本两种输入条件下、三种提示策略的表现。
- 三种提示策略包括无系统提示的基础提示、专科角色原始提示及链式思考提示。
- 诊断表现通过多选题和鉴别诊断输出评估,资源消耗按每个病例各场景的token消耗量计算。
诊断标准与关键数值
- 纯影像输入、链式思考提示下,o1多选题准确率为56.2%,高于人类基准55.9%。
- 纯影像输入、链式思考提示下,Claude-3.7-Sonnet多选题准确率为49.3%。
- 纯影像输入、链式思考提示下,Gemini-2.0-Flash-Thinking-Experimental多选题准确率为37%。
- 加入描述文本后,o1在基础与原始提示下取得最高准确率71.2%。
- 原始提示在o1纯影像加鉴别诊断条件下输出token少于基础与链式思考提示,所有p均小于0.01。
临床应用与阅片要点
- 模型输出token消耗不能作为判断诊断准确的可靠指标。
- 描述文本输入可提高模型诊断表现,提示在影像报告中提供规范描述可能有助于模型应用。
局限性
- 病例来源于单一学会网站,可能影响任务代表性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: We evaluated the diagnostic performance and resource efficiency of three multimodal-reasoning-models for radiological image interpretation.
METHODS: Using three multimodal-reasoning-models, we analyzed 73 cases under different conditions (Imaging-Only and Combined-Descriptive-Text) with three system prompt types (basic [without system prompt], original [specialized-role], and chain-of-thought [CoT] prompts). Quiz cases were extracted from the Korean Society of Ultrasound in Medicine Website, along with corresponding human benchmark data. Diagnostic performance was assessed through Multiple-Choice (MCQ) and Differential-Diagnosis (DDx) outputs. Resource utilization was measured by token consumption for each case across all scenarios. Pearson correlation coefficients were calculated to evaluate associations between token usage and diagnostic accuracy.
RESULTS: For imaging-only input, under CoT prompt, o1 demonstrated superior accuracy of 56.2 %, surpassing the 55.9 % human benchmark compared to Claude-3.7-Sonnet (49.3 %) and Gemini-2.0-Flash-Thinking-Experimental (37 %) for MCQ. The integration of descriptive-text inputs substantially increased performance across all models, with o1 achieving the highest accuracy (71.2 %, with basic and original). This performance advantage was most pronounced in DDx. Original prompts utilized fewer output tokens while maintaining comparable accuracy for o1 (Imaging-Only with DDx: original vs. basic, CoT prompts, all p < 0.01). Intra-model analysis revealed a negative correlation between accuracy and output token for o1 (r = -0.41), while inter-model analysis showed strong positive correlations between total token and accuracy (r = 0.93 for Imaging-Only with MCQ).
CONCLUSION: The paradoxical relationship between resource utilization and diagnostic accuracy suggests that model architecture fundamentally determines baseline performance, while prompt optimization influences efficiency within architectural constraints on multimodal-reasoning-models.
引用
- Han, Lee, Jeong, et al. Relationship between resource utilization and diagnostic accuracy of large language models for efficient multimodal reasoning in radiologic image interpretation. Eur J Radiol. 2026. PMID: 41576425. doi:10.1016/j.ejrad.2026.112677
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。