文献速递头颈外科

评估革命性人工智能聊天机器人口咽癌信息的准确性

Evaluation of Oropharyngeal Cancer Information from Revolutionary Artificial Intelligence Chatbot.
PubMed 文献 · Laryngoscope · 2023-11-20

研究将15个口咽癌相关问题输入ChatGPT 3.5版,由4名医生独立评估准确性、全面性和与医生回答的相似性(5分Likert量表)。平均准确性3.88(SD=0.99)、全面性3.80(SD=1.14)、相似性3.67(SD=1.08);治疗后相关问题在三个维度均显著高于诊断相关问题(p<0.01)。可读性高于11年级水平,高于推荐的患者6年级水平。结论认为ChatGPT回答可能无法适当教育患者、可能误导患者且阅读难度高于推荐水平。

要点

  • ChatGPT口咽癌回答平均准确性3.88、全面性3.80、与医生回答相似性3.67(5分制)。
  • 治疗后相关问题在准确性、全面性、相似性三个维度均显著高于诊断相关问题(p<0.01)。
  • 平均FRE和FKRGL评分均提示可读性高于11年级水平,高于推荐的6年级水平。

核心结论

  • ChatGPT回答可能无法将患者教育到适当程度,可能 outright 误导患者,且阅读难度高于患者材料推荐水平。
  • 口咽癌患者面临复杂、改变生活的诊断和治疗,属于脆弱人群,在消费聊天机器人生成的医疗信息时应谨慎。

研究设计与样本

  • 研究开发了15个口咽癌相关问题并输入ChatGPT 3.5版。
  • 4名医生评分者使用5分Likert量表独立评估准确性、全面性和与医生回答的相似性。
  • 评分低于3分的回答由医生评分者进行批评,批评采用归纳主题分析。
  • 回答可读性使用Flesch阅读易读性(FRE)和Flesch-Kincaid阅读年级水平(FKRGL)量表评估。

主要发现

  • 平均准确性评分为3.88(SD=0.99)。
  • 平均全面性评分为3.80(SD=1.14)。
  • 平均与医生回答相似性评分为3.67(SD=1.08)。
  • 治疗后相关问题在准确性、全面性和相似性三个维度均显著高于诊断相关问题(p<0.01)。
  • 医生批评识别出两个主题:教育价值欠佳和可能误导患者。

关键数值与推荐

  • 准确性、全面性、相似性评分采用5分Likert量表,评分低于3分的回答被批评。
  • 治疗后相关问题在三个维度均显著高于诊断相关问题(p<0.01)。
  • 平均FRE和FKRGL评分均高于11年级可读性水平,高于推荐的6年级水平。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: With burgeoning popularity of artificial intelligence-based chatbots, oropharyngeal cancer patients now have access to a novel source of medical information. Because chatbot information is not reviewed by experts, we sought to evaluate an artificial intelligence-based chatbot's oropharyngeal cancer-related information for accuracy. METHODS: Fifteen oropharyngeal cancer-related questions were developed and input into ChatGPT version 3.5. Four physician-graders independently assessed accuracy, comprehensiveness, and similarity to a physician response using 5-point Likert scales. Responses graded lower than three were then critiqued by physician-graders. Critiques were analyzed using inductive thematic analysis. Readability of responses was assessed using Flesch Reading Ease (FRE) and Flesch-Kincaid Reading Grade Level (FKRGL) scales. RESULTS: Average accuracy, comprehensiveness, and similarity to a physician response scores were 3.88 (SD = 0.99), 3.80 (SD = 1.14), and 3.67 (SD = 1.08), respectively. Posttreatment-related questions were most accurate, comprehensive, and similar to a physician response, followed by treatment-related, then diagnosis-related questions. Posttreatment-related questions scored significantly higher than diagnosis-related questions in all three domains (p < 0.01). Two themes of the physician critiques were identified: suboptimal education value and potential to misinform patients. The mean FRE and FKRGL scores both indicated greater than an 11th grade readability level-higher than the 6th grade level recommended for patients. CONCLUSION: ChatGPT responses may not educate patients to an appropriate degree, could outright misinform them, and read at a more difficult grade level than is recommended for patient material. As oropharyngeal cancer patients represent a vulnerable population facing complex, life-altering diagnoses, and treatments, they should be cautious when consuming chatbot-generated medical information. LEVEL OF EVIDENCE: NA Laryngoscope, 134:2252-2257, 2024.

来源:Laryngoscope,查看原文。版权归原作者所有。

引用

  1. Davis RJ, Ayo-Ajibola O, Lin ME, et al. Evaluation of Oropharyngeal Cancer Information from Revolutionary Artificial Intelligence Chatbot. Laryngoscope. 2023. PMID: 37983846. doi:10.1002/lary.31191

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。