4种人工智能聊天机器人在回答牙髓病学问题中的表现
本研究评估ChatGPT-4o、MedGebra GPT-4o、Meta LIama 3和Gemini Advanced回答100道牙髓病学多选题的表现,间隔1周测试两次。MedGebra GPT-4o第一轮正确率最高(93%),ChatGPT-4o第二轮最高(90%),Meta Llama 3两轮最低(73%和75%)。结论认为需进一步训练开发以提高准确性和可靠性。
要点
- MedGebra GPT-4o第一轮正确率93%,ChatGPT-4o第二轮正确率90%。
- Meta Llama 3两轮正确率最低,分别为73%和75%。
- ChatGPT-4o表现稳定性最高,其次为Gemini Advanced和Meta Llama 3。
核心结论
- ChatGPT-4o和MedGebra GPT-4o回答牙髓病学多选题的正确率较高,Meta LIama 3和Gemini Advanced表现较低。
- 需要进一步训练和开发以提高这些聊天机器人在牙髓病学中的准确性和可靠性。
研究设计与样本
- 研究使用100道牙髓病学多选题,每题有4个备选答案,题目来自2本知名牙髓病学教科书。
- 评估ChatGPT-4o、MedGebra GPT-4o、Meta LIama 3和Gemini Advanced选择正确答案的表现,间隔1周测试两次。
主要发现
- 两轮表现稳定性最高为ChatGPT-4o,其次为Gemini Advanced和Meta Llama 3。
- MedGebra GPT-4o第一轮正确回答百分比最高(93%),ChatGPT-4o第二轮最高(90%)。
- Meta Llama 3第一轮正确回答百分比最低(73%),第二轮最低(75%)。
- MedGebra GPT-4o第一轮表现最佳,但第二轮稳定性较低(McNemar P > .05;Kappa = 0.725,P < .001)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: Artificial intelligence models have shown potential as educational tools in healthcare, such as answering exam questions. This study aimed to assess the performance of 4 prominent chatbots: ChatGPT-4o, MedGebra GPT-4o, Meta LIama 3, and Gemini Advanced in answering multiple-choice questions (MCQs) in endodontics. METHODS: The study utilized 100 MCQs, each with 4 potential answers. These MCQs were obtained from 2 well-known endodontic textbooks. The performance of the above chatbots regarding choosing the correct answers was assessed twice with a 1-week interval. RESULTS: The stability of the performance in the 2 rounds was highest for ChatGPT-4o, followed by Gemini Advanced and Meta Llama 3. MedGebra GPT-4o provided the highest percentage of true answers in the first round (93%) followed by ChatGPT-4o in the second round (90%). Meta Llama 3 provided the lowest percentages in the first (73%) and second rounds (75%). Although the performance of MedGebra GPT-4o was the best in the first round, it was less stable upon the second round (McNemar P > .05; Kappa = 0.725, P < .001). CONCLUSIONS: ChatGPT-4o and MedGebra GPT-4o answered a high fraction of endodontic MCQs, while Meta LIama 3 and Gemini Advanced showed lower performance. Further training and development are required to improve their accuracy and reliability in endodontics.
引用
- Abdulrab S, Abada H, Mashyakhy M, et al. Performance of 4 Artificial Intelligence Chatbots in Answering Endodontic Questions. J Endod. 2025. PMID: 39814135. doi:10.1016/j.joen.2025.01.002
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。