7种人工智能聊天机器人在牙髓病学考试题中的表现
本研究评估了7种人工智能聊天机器人在回答美国牙髓病学委员会笔试风格选择题时的表现。结果显示准确率从48%(Microsoft Copilot)到71%(Gemini Advanced、GPT-3.5和Claude 3.5 Sonnet)不等(P < .05),且部分聊天机器人在教材来源题目上表现更优(P < .05)。推理质量评分分布因聊天机器人而异(P < .05)。
要点
- 各聊天机器人准确率48%~71%,Gemini Advanced、GPT-3.5和Claude 3.5 Sonnet并列最高(71%)。
- GPT-3.5、GPT-4o、GPT-4.0和Claude 3.5 Sonnet在教材来源题目上表现显著更好(P < .05)。
- Gemini Advanced的推理质量评分2占比最高(81%),评分0占比最低(18.5%)。
核心结论
- 综合评估显示7种人工智能聊天机器人在牙髓病学考试题上各有优势与局限,可作为教育参考资源。
研究设计与样本
- 由两名委员会认证牙髓病学家生成100道美国牙髓病学委员会笔试风格选择题,提交给7种聊天机器人各3次,仅3次均正确计为正确;两名校准评审独立评分2100条回答。
主要发现
- 准确率范围为48%(Microsoft Copilot)至71%(Gemini Advanced、GPT-3.5和Claude 3.5 Sonnet)(P < .05)。
- Gemini Advanced、Gemini和Microsoft Copilot在不同题目来源间表现相似(P > .05)。
- GPT-3.5、GPT-4o、GPT-4.0和Claude 3.5 Sonnet在教材来源题目上表现显著更优(P < .05)。
- 推理评分分布在各聊天机器人间存在差异(P < .05)。
- Gemini Advanced的评分2比例最高(81%),评分0比例最低(18.5%)。
关键数值与推荐
- 准确率定义为3次尝试均选择正确答案。
- 推理质量采用三等级量表(0、1、2)评分。
临床怎么用
- 牙髓病学教育中可参考聊天机器人的准确率与推理质量,但需注意其对题目来源的敏感性及局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: The aim of this study was to assess the overall performance of artificial intelligence chatbots in answering board-style endodontic questions. METHODS: One hundred multiple choice endodontic questions, following the style of American Board of Endodontics Written Exam, were generated by two board-certified endodontists. The questions were submitted to the following chatbots, three times in a row: Gemini Advanced, Gemini, Microsoft Copilot, GPT-3.5, GPT-4o, GPT-4.0, and Claude 3.5 Sonnet. The chatbot was asked to choose the correct response and to explain the justification. The response to the question was considered "correct" only if the chatbot picked the right choice in ALL 3 attempts. The quality of reasoning as to why the chatbot selected the answer choice was scored using a three-ordinal scale (0, 1, 2). Two calibrated reviewers scored all 2100 responses independently. Categorical data were analyzed using Chi-square test; ordinal data were analyzed using Kruskal-Wallis and Mann-Whitney tests. RESULTS: The accuracy scores ranged from 48% (Microsoft Copilot) to 71% (Gemini Advanced, GPT-3.5, and Claude 3.5 Sonnet) (P < .05). Gemini Advanced, Gemini, and Microsoft Copilot showed similar performance regardless of the question source (textbook or literature) (P > .05). GPT-3.5, GPT-4o, GPT-4.0 and Claude 3.5 Sonnet performed significantly better with textbook-based questions (P < .05). Reasoning scores showed different distribution among chatbots (P < .05). Gemini Advanced had the highest rate of score 2 (81%) and the lowest rate of score 0 (18.5%). CONCLUSIONS: Comprehensive assessment of seven AI chatbots' performance on board-style endodontic questions revealed their capacities and limitations as educational resources in the field of endodontics.
引用
- Jalali P, Mohammad-Rahimi H, Wang FM, et al. Performance of 7 Artificial Intelligence Chatbots on Board-style Endodontic Questions. J Endod. 2025. PMID: 40581328. doi:10.1016/j.joen.2025.06.014
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。