文献速递鼻科小儿耳鼻咽喉

评估ChatGPT在回答过敏性鼻炎与慢性鼻窦炎相关问题中的表现

Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis.
PubMed 文献 · Otolaryngol Head Neck Surg · 2024-05-26

这项横断面研究由2名资深鼻科教授独立评分,评估GPT-3.5和GPT-4对过敏性鼻炎(n=189)和慢性鼻窦炎(n=242)问题的回答准确性。ChatGPT整体表现满意,所有类别中准确回答超过80%的问题;GPT-4.0回答过敏性鼻炎问题的准确性显著高于GPT-3.5,但慢性鼻窦炎问题中无此差异。

要点

  • ChatGPT在所有类别中准确回答超过80%的问题。
  • GPT-4.0回答过敏性鼻炎问题的准确性显著高于GPT-3.5。
  • ChatGPT在近期进展类问题上表现欠佳,中文提问不影响准确性。

核心结论

  • ChatGPT能够为过敏性鼻炎和慢性鼻窦炎患者传达准确信息,其在不同领域的表现为其应用和改进提供了参考。

研究设计与样本

  • 横断面研究,每个问题作为独立提示输入,由2名资深鼻科教授独立评分,分歧由第三名评审裁定。
  • 评估过敏性鼻炎问题n=189和慢性鼻窦炎问题n=242,由GPT-3.5和GPT-4生成回答。

主要发现

  • ChatGPT整体表现满意,所有类别中准确回答超过80%的问题。
  • GPT-4.0回答过敏性鼻炎问题的准确性显著高于GPT-3.5,但慢性鼻窦炎问题中无此差异。
  • 使用GPT-4.0回答过敏性鼻炎问题时,患者来源问题的准确性显著高于医生来源问题;GPT-3.5或慢性鼻窦炎问题中未观察到该差异。
  • ChatGPT在过敏性鼻炎和慢性鼻窦炎的基础知识、预防和情感内容方面表现优异,但在近期进展类问题上存在挑战,GPT-3.5和GPT-4.0均如此。
  • 中文提问时回答准确性不受影响。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: This study aims to evaluate the accuracy of ChatGPT in answering allergic rhinitis (AR) and chronic rhinosinusitis (CRS) related questions. STUDY DESIGN: This is a cross-sectional study. SETTING: Each question was inputted as a separate, independent prompt. METHODS: Responses to AR (n = 189) and CRS (n = 242) related questions, generated by GPT-3.5 and GPT-4, were independently graded for accuracy by 2 senior rhinology professors, with disagreements adjudicated by a third reviewer. RESULTS: Overall, ChatGPT demonstrated a satisfactory performance, accurately answering over 80% of questions across all categories. Specifically, GPT-4.0's accuracy in responding to AR-related questions significantly exceeded that of GPT-3.5, but distinction not evident in CRS-related questions. Patient-originated questions had a significantly higher accuracy compared to doctor-originated questions when utilizing GPT-4.0 to respond to AR-related questions. This discrepancy was not observed with GPT-3.5 or in the context of CRS-related questions. Across different types of content, ChatGPT excelled in covering basic knowledge, prevention, and emotion for AR and CRS. However, it experienced challenges when addressing questions about recent advancements, a trend consistent across both GPT-3.5 and GPT-4.0 iterations. Importantly, the accuracy of responses remained unaffected when questions were posed in Chinese. CONCLUSION: Our findings suggest ChatGPT's capability to convey accurate information for AR and CRS patients, and offer insights into its performance across various domains, guiding its utilization and improvement.

来源:Otolaryngol Head Neck Surg,查看原文。版权归原作者所有。

引用

  1. Ye F, Zhang H, Luo X, et al. Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis. Otolaryngol Head Neck Surg. 2024. PMID: 38796735. doi:10.1002/ohn.832

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。