文献速递鼻科

常用鼻科疾病聊天机器人回答的定量比较

Quantitative Comparison of Chatbots on Common Rhinology Pathologies.
PubMed 文献 · Laryngoscope · 2024-04-26

2023年5月向ChatGPT、ChatGPT Plus、Google Bard和Microsoft Bing询问六种常见鼻科疾病的治疗,由评审者用经验证量表评估可读性、质量、可理解性和可操作性,并由两名鼻科专家评估准确性和全面性,结果提示ChatGPT Plus和Google Bard总体最便于患者使用。

要点

  • Bard与Bing的FRE可读性评分分别为46.50和46.32,高于ChatGPT的33.17和Plus的35.93。
  • ChatGPT、Plus、Bard的DISCERN质量评分分别为20.42、20.89、20.61,均高于Bing的16.97(p<0.001)。
  • ChatGPT Plus准确性评分4.39,高于ChatGPT的3.97、Bard的3.72和Bing的3.19。

核心结论

  • 在测试领域的综合表现上,结果提示ChatGPT Plus和Google Bard是目前治疗常见鼻科疾病时对患者最友好的聊天机器人。

研究设计与样本

  • 2023年5月向ChatGPT、ChatGPT Plus、Google Bard和Microsoft Bing询问鼻出血、慢性鼻窦炎、鼻窦感染、过敏性鼻炎、过敏和鼻息肉的治疗。
  • 由评审者使用经验证评分指标分析各回答的可读性、质量、可理解性和可操作性,并由两名鼻科专家评估准确性和全面性。

主要发现

  • ChatGPT、Plus、Bard、Bing的FRE可读性评分分别为33.17、35.93、46.50、46.32,Bard较ChatGPT高(p=0.003)、较Plus高(p=0.025),Bing较ChatGPT高(p=0.008)、较Plus高(p=0.048)。
  • ChatGPT、Plus、Bard的DISCERN质量评分分别为20.42、20.89、20.61,均高于Bing的16.97(p<0.001)。
  • ChatGPT和Bing的PEMAT可理解性评分分别为76.67和66.61,均低于Plus的92.00(p<0.001,p<0.001)和Bard的92.67(p<0.001,p<0.001)。
  • ChatGPT Plus的准确性评分为4.39,高于ChatGPT的3.97(p=0.118)、Bard的3.72(p=0.002)和Bing的3.19(p<0.001)。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: Understanding the strengths and weaknesses of chatbots as a source of patient information is critical for providers in the rising artificial intelligence landscape. This study is the first to quantitatively analyze and compare four of the most used chatbots available regarding treatments of common pathologies in rhinology. METHODS: The treatment of epistaxis, chronic sinusitis, sinus infection, allergic rhinitis, allergies, and nasal polyps was asked to chatbots ChatGPT, ChatGPT Plus, Google Bard, and Microsoft Bing in May 2023. Individual responses were analyzed by reviewers for readability, quality, understandability, and actionability using validated scoring metrics. Accuracy and comprehensiveness were evaluated for each response by two experts in rhinology. RESULTS: ChatGPT, Plus, Bard, and Bing had FRE readability scores of 33.17, 35.93, 46.50, and 46.32, respectively, indicating higher readability for Bard and Bing compared to ChatGPT (p = 0.003, p = 0.008) and Plus (p = 0.025, p = 0.048). ChatGPT, Plus, and Bard had mean DISCERN quality scores of 20.42, 20.89, and 20.61, respectively, which was higher than the score for Bing of 16.97 (p < 0.001). For understandability, ChatGPT and Bing had PEMAT scores of 76.67 and 66.61, respectively, which were lower than both Plus at 92.00 (p < 0.001, p < 0.001) and Bard at 92.67 (p < 0.001, p < 0.001). ChatGPT Plus had an accuracy score of 4.39 which was higher than ChatGPT (3.97, p = 0.118), Bard (3.72, p = 0.002), and Bing (3.19, p < 0.001). CONCLUSION: On aggregate of the tested domains, our results suggest ChatGPT Plus and Google Bard are currently the most patient-friendly chatbots for the treatment of common pathologies in rhinology. LEVEL OF EVIDENCE: N/A Laryngoscope, 134:4225-4231, 2024.

来源:Laryngoscope,查看原文。版权归原作者所有。

引用

  1. Bellinger JR, Kwak MW, Ramos GA, et al. Quantitative Comparison of Chatbots on Common Rhinology Pathologies. Laryngoscope. 2024. PMID: 38666768. doi:10.1002/lary.31470

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。