评估ChatGPT在回答过敏性鼻炎与慢性鼻窦炎相关问题中的表现
这项横断面研究由2名资深鼻科教授独立评分,评估GPT-3.5和GPT-4对过敏性鼻炎(n=189)和慢性鼻窦炎(n=242)问题的回答准确性。ChatGPT整体表现满意,所有类别中准确回答超过80%的问题;GPT-4.0回答过敏性鼻炎问题的准确性显著高于GPT-3.5,但慢性鼻窦炎问题中无此差异。
要点
- ChatGPT在所有类别中准确回答超过80%的问题。
- GPT-4.0回答过敏性鼻炎问题的准确性显著高于GPT-3.5。
- ChatGPT在近期进展类问题上表现欠佳,中文提问不影响准确性。
核心结论
- ChatGPT能够为过敏性鼻炎和慢性鼻窦炎患者传达准确信息,其在不同领域的表现为其应用和改进提供了参考。
研究设计与样本
- 横断面研究,每个问题作为独立提示输入,由2名资深鼻科教授独立评分,分歧由第三名评审裁定。
- 评估过敏性鼻炎问题n=189和慢性鼻窦炎问题n=242,由GPT-3.5和GPT-4生成回答。
主要发现
- ChatGPT整体表现满意,所有类别中准确回答超过80%的问题。
- GPT-4.0回答过敏性鼻炎问题的准确性显著高于GPT-3.5,但慢性鼻窦炎问题中无此差异。
- 使用GPT-4.0回答过敏性鼻炎问题时,患者来源问题的准确性显著高于医生来源问题;GPT-3.5或慢性鼻窦炎问题中未观察到该差异。
- ChatGPT在过敏性鼻炎和慢性鼻窦炎的基础知识、预防和情感内容方面表现优异,但在近期进展类问题上存在挑战,GPT-3.5和GPT-4.0均如此。
- 中文提问时回答准确性不受影响。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: This study aims to evaluate the accuracy of ChatGPT in answering allergic rhinitis (AR) and chronic rhinosinusitis (CRS) related questions. STUDY DESIGN: This is a cross-sectional study. SETTING: Each question was inputted as a separate, independent prompt. METHODS: Responses to AR (n = 189) and CRS (n = 242) related questions, generated by GPT-3.5 and GPT-4, were independently graded for accuracy by 2 senior rhinology professors, with disagreements adjudicated by a third reviewer. RESULTS: Overall, ChatGPT demonstrated a satisfactory performance, accurately answering over 80% of questions across all categories. Specifically, GPT-4.0's accuracy in responding to AR-related questions significantly exceeded that of GPT-3.5, but distinction not evident in CRS-related questions. Patient-originated questions had a significantly higher accuracy compared to doctor-originated questions when utilizing GPT-4.0 to respond to AR-related questions. This discrepancy was not observed with GPT-3.5 or in the context of CRS-related questions. Across different types of content, ChatGPT excelled in covering basic knowledge, prevention, and emotion for AR and CRS. However, it experienced challenges when addressing questions about recent advancements, a trend consistent across both GPT-3.5 and GPT-4.0 iterations. Importantly, the accuracy of responses remained unaffected when questions were posed in Chinese. CONCLUSION: Our findings suggest ChatGPT's capability to convey accurate information for AR and CRS patients, and offer insights into its performance across various domains, guiding its utilization and improvement.
引用
- Ye F, Zhang H, Luo X, et al. Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis. Otolaryngol Head Neck Surg. 2024. PMID: 38796735. doi:10.1002/ohn.832
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。