超越考试准确率:追踪持续失败集揭示多模态大语言模型的视觉牙科推理缺陷
该研究于2026年1月对GPT-5.2T、Claude 4.5和Gemini 3进行零样本测试,使用350道日本国家牙科考试2025年题目和33道持续失败集题目。结果显示模型在纯文本题上准确率高,但视觉题准确率明显较低,且部分视觉题所有模型均答错,提示视觉牙科推理存在显著差距。
要点
- 所有模型均答错的21题中16题为视觉题,集中在儿童口腔和正畸领域。
- 持续失败集中9/33题所有模型仍答错,5/33题所有模型均答对。
核心结论
- 执照考试基准测试显示纯文本题接近天花板,但视觉牙科推理仍存在显著差距。
- 总体考试准确率高不能作为判断图像条件下牙科支持可靠性的充分证据。
研究设计与样本
- 于2026年1月采用零样本协议(无提示工程)测试GPT-5.2T、Claude 4.5和Gemini 3。
- 使用350道公开的日本国家牙科考试2025年题目(202道纯文本题,148道视觉题)和33道持续失败集题目。
- 持续失败集为先前JNDE-2024评估中所有模型均答错的题目。
- 根据官方答案键评分,使用Cochran's Q检验及Bonferroni校正的成对比较。
主要发现
- 所有模型总体准确率较高,但视觉题准确率(71.0-79.7%)低于纯文本题(91.6-95.5%)。
- Gemini 3总体准确率最高(88.9%,311/350),其次GPT-5.2T(84.3%,295/350)和Claude 4.5(84.0%,294/350)。
- 21题(6.0%)所有模型均答错,其中16/21为视觉题,集中在儿童口腔和正畸领域。
- 持续失败集中5/33题所有模型均答对,9/33题所有模型仍答错。
关键数值与推荐
- 视觉题准确率范围71.0-79.7%,纯文本题91.6-95.5%。
- Gemini 3总体准确率88.9%(311/350),GPT-5.2T 84.3%(295/350),Claude 4.5 84.0%(294/350)。
- 所有模型均答错的题目占6.0%(21/350),其中视觉题占16/21。
- 持续失败集中所有模型答对5/33,仍答错9/33。
临床怎么用
- 临床评估多模态大语言模型时应按模态分层报告,并关注视觉推理能力。
- 不应仅凭总体考试准确率高就认为模型在图像条件下可靠。
- 需使用牙科专用挑战集跟踪模型进展,超越总体考试准确率。
局限性
- 原文未报告
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To benchmark late-2025 general-purpose multimodal large language models (LLMs) on the Japanese National Dental Examination (JNDE) and to reassess a previously identified persistent-failure set. METHODS: GPT-5.2T, Claude 4.5, and Gemini 3 were tested in January 2026 using a zero-shot protocol (no prompt engineering) on 350 publicly released JNDE-2025 questions (202 text-only; 148 visually-based) and on 33 questions in a persistent-failure set that had been answered incorrectly by all models in a prior JNDE-2024 evaluation. Responses were scored against the official answer key; paired accuracies were compared using Cochran's Q test followed by post hoc pairwise comparisons with Bonferroni-adjusted p values. RESULTS: Overall accuracy was high across all three models, but each model performed worse on visually-based than on text-only questions (71.0-79.7 % vs 91.6-95.5 %). Gemini 3 achieved the highest overall accuracy (88.9 %, 311/350), followed by GPT-5.2T (84.3 %, 295/350) and Claude 4.5 (84.0 %, 294/350). Twenty-one questions (6.0 %) were missed by all models and were predominantly visually-based (16/21), clustering in pediatric dentistry and orthodontics. In the persistent-failure set, 5/33 questions were answered correctly by all models, whereas 9/33 remained incorrect across all models. CONCLUSIONS: Licensing-examination benchmarking yields near-ceiling performance on text-only questions, but substantial gaps remain in visual dental reasoning. Whole-examination benchmarking should therefore be complemented by modality-stratified reporting and dentistry-specific challenge sets to track meaningful progress beyond aggregate examination accuracy. CLINICAL SIGNIFICANCE: High aggregate examination accuracy should not be interpreted as sufficient evidence to judge the reliability of image-conditioned dental support; more clinically grounded evaluations are required.
引用
- Mine Y, Taji T, Okazaki S, et al. Beyond exam accuracy: Tracking a persistent-failure set reveals visual dental reasoning gaps in multimodal LLMs. J Dent. 2026. PMID: 41946430. doi:10.1016/j.jdent.2026.106675
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。