聊天机器人与临床医生回答儿童口腔问题的准确性与一致性:一项初步研究
本研究比较了聊天机器人与临床医生回答儿童口腔常见问题的准确性与一致性。结果显示,儿童口腔专科医生的准确性(均值±标准差 96.67%±4.3%)显著高于其他临床医生和聊天机器人(P<0.001),聊天机器人中ChatGPT准确性最高(78%±3%)。结论认为聊天机器人准确性低于牙医,尚不推荐用于临床儿童口腔。
要点
- 儿童口腔专科医生回答准确性最高(96.67%±4.3%),显著优于聊天机器人(P<0.001)。
- 聊天机器人中ChatGPT准确性最高(78%±3%),但所有聊天机器人准确性均低于牙医。
- 除ChatGPT3.5外,其他聊天机器人一致性可接受(Cronbach alpha>0.7)。
核心结论
- 在本初步研究中,聊天机器人的准确性低于牙医,尚不推荐用于临床儿童口腔。
- 聊天机器人可能作为教育和患者信息传播的辅助工具,但尚不能替代人类临床医生进行诊断决策。
研究设计与样本
- 使用9种公开可用的聊天机器人(Google Bard、ChatGPT4、ChatGPT 3.5、Llama、Sage、Claude 2 100k、Claude-instant、Claude-instant-100k、Google Palm)回答问题,每种进行3次独立新对话。
- 三组临床医生(全科牙医、儿童口腔专科医生、学生;每组n=20)也回答了问题。
- 由两名儿童口腔科教师及一名独立儿童口腔牙医对回答进行评分。
- 使用方差分析(ANOVA)比较准确性(正确回答百分比),事后两两比较采用Tukey HSD法校正;计算Cronbach alpha确定一致性。
主要发现
- 儿童口腔专科医生准确性(均值±标准差 96.67%±4.3%)显著高于其他临床医生和聊天机器人(P<0.001)。
- 全科牙医准确性(88.0%±6.1%)显著高于聊天机器人(P<0.001),学生准确性为80.8%±6.9%。
- 聊天机器人中ChatGPT准确性最高(78%±3%)。
- 除ChatGPT3.5外,所有聊天机器人均显示可接受的一致性(Cronbach alpha>0.7)。
关键数值与推荐
- 儿童口腔专科医生准确性均值为96.67%(标准差4.3%)。
- 全科牙医准确性均值为88.0%(标准差6.1%)。
- 学生准确性均值为80.8%(标准差6.9%)。
- ChatGPT准确性均值为78%(标准差3%)。
- 除ChatGPT3.5外,聊天机器人Cronbach alpha>0.7。
临床怎么用
- 聊天机器人可作为教育和患者信息传播的辅助工具。
- 聊天机器人尚不能替代人类临床医生进行诊断决策。
- 临床儿童口腔实践中暂不推荐使用聊天机器人。
局限性
- 本研究为初步研究(pilot study)。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: Artificial Intelligence has applications such as Large Language Models (LLMs), which simulate human-like conversations. The potential of LLMs in healthcare is not fully evaluated. This pilot study assessed the accuracy and consistency of chatbots and clinicians in answering common questions in pediatric dentistry. METHODS: Two expert pediatric dentists developed thirty true or false questions involving different aspects of pediatric dentistry. Publicly accessible chatbots (Google Bard, ChatGPT4, ChatGPT 3.5, Llama, Sage, Claude 2 100k, Claude-instant, Claude-instant-100k, and Google Palm) were employed to answer the questions (3 independent new conversations). Three groups of clinicians (general dentists, pediatric specialists, and students; n = 20/group) also answered. Responses were graded by two pediatric dentistry faculty members, along with a third independent pediatric dentist. Resulting accuracies (percentage of correct responses) were compared using analysis of variance (ANOVA), and post-hoc pairwise group comparisons were corrected using Tukey's HSD method. ACronbach's alpha was calculated to determine consistency. RESULTS: Pediatric dentists were significantly more accurate (mean±SD 96.67 %± 4.3 %) than other clinicians and chatbots (p < 0.001). General dentists (88.0 % ± 6.1 %) also demonstrated significantly higher accuracy than chatbots (p < 0.001), followed by students (80.8 %±6.9 %). ChatGPT showed the highest accuracy (78 %±3 %) among chatbots. All chatbots except ChatGPT3.5 showed acceptable consistency (Cronbach alpha>0.7). CLINICAL SIGNIFICANCE: Based on this pilot study, chatbots may be valuable adjuncts for educational purposes and for distributing information to patients. However, they are not yet ready to serve as substitutes for human clinicians in diagnostic decision-making. CONCLUSION: In this pilot study, chatbots showed lower accuracy than dentists. Chatbots may not yet be recommended for clinical pediatric dentistry.
引用
- Rokhshad R, Zhang P, Mohammad-Rahimi H, et al. Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: A pilot study. J Dent. 2024. PMID: 38499280. doi:10.1016/j.jdent.2024.104938
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。