文献速递牙体牙髓病

人工智能聊天机器人作为牙体牙髓病公共信息源的有效性与可靠性

Validity and reliability of artificial intelligence chatbots as public sources of information on endodontics.
PubMed 文献 · Int Endod J · 2023-12-20

本研究由牙体牙髓病专家制定常见问题,分别向GPT-3.5、Google Bard和Bing各提问三次,共获得180条回答,由两名专科医生采用改良全球质量评分独立评估。结果显示GPT-3.5在高阈值下的有效性显著高于Bard和Bing,三者可靠性均可接受。

要点

  • GPT-3.5在高阈值有效性上显著优于Google Bard和Bing(p=.008)。
  • 低阈值下三平台有效性无显著差异(p>.05)。
  • 三种聊天机器人可靠性均达可接受水平(Cronbach's alpha>0.7)。

核心结论

  • GPT-3.5在牙体牙髓病相关话题上提供的信息比Google Bard和Bing更可信。

研究设计与样本

  • 由10名牙体牙髓病专家制定常见问题,并通过GPT-3.5查询收集10个问题,每个问题向每个聊天机器人提问三次,共获得180条回答。
  • 两名委员会认证的牙体牙髓病医生使用改良全球质量评分(5点Likert量表)独立评估回答。

主要发现

  • 低阈值有效性(三次回答均≥4分):GPT-3.5为95%,Google Bard为85%,Bing为75%,平台间无显著差异(p>.05)。
  • 高阈值有效性(三次回答均为5分):GPT-3.5为60%,Google Bard为15%,Bing为15%,GPT-3.5显著高于Bard和Bing(p=.008)。
  • 所有聊天机器人的Cronbach's alpha均>0.7,达到可接受的可靠性水平。

关键数值与推荐

  • 低阈值有效性定义为三次回答评分均≥4分,高阈值有效性定义为三次回答评分均为5分。

临床怎么用

  • 牙体牙髓病相关信息查询可优先考虑GPT-3.5,但高阈值下其有效性仅60%,仍需专科医生审核。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

AIM: This study aimed to evaluate and compare the validity and reliability of responses provided by GPT-3.5, Google Bard, and Bing to frequently asked questions (FAQs) in the field of endodontics. METHODOLOGY: FAQs were formulated by expert endodontists (n = 10) and collected through GPT-3.5 queries (n = 10), with every question posed to each chatbot three times. Responses (N = 180) were independently evaluated by two board-certified endodontists using a modified Global Quality Score (GQS) on a 5-point Likert scale (5: strongly agree; 4: agree; 3: neutral; 2: disagree; 1: strongly disagree). Disagreements on scoring were resolved through evidence-based discussions. The validity of responses was analysed by categorizing scores into valid or invalid at two thresholds: The low threshold was set at score ≥4 for all three responses whilst the high threshold was set at score 5 for all three responses. Fisher's exact test was conducted to compare the validity of responses between chatbots. Cronbach's alpha was calculated to assess the reliability by assessing the consistency of repeated responses for each chatbot. RESULTS: All three chatbots provided answers to all questions. Using the low-threshold validity test (GPT-3.5: 95%; Google Bard: 85%; Bing: 75%), there was no significant difference between the platforms (p > .05). When using the high-threshold validity test, the chatbot scores were substantially lower (GPT-3.5: 60%; Google Bard: 15%; Bing: 15%). The validity of GPT-3.5 responses was significantly higher than Google Bard and Bing (p = .008). All three chatbots achieved an acceptable level of reliability (Cronbach's alpha >0.7). CONCLUSIONS: GPT-3.5 provided more credible information on topics related to endodontics compared to Google Bard and Bing.

来源:Int Endod J,查看原文。版权归原作者所有。

引用

  1. Mohammad-Rahimi H, Ourang SA, Pourhoseingholi MA, et al. Validity and reliability of artificial intelligence chatbots as public sources of information on endodontics. Int Endod J. 2023. PMID: 38117284. doi:10.1111/iej.14014

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。