生成式与检索式聊天机器人在回答年龄相关性黄斑变性和糖尿病视网膜病变患者问题上的比较
该横断面研究比较了生成式模型(ChatGPT-4、ChatGPT-3.5、Google Bard)与检索式模型(OcularBERT)回答45个年龄相关性黄斑变性、糖尿病视网膜病变及其他眼科问题的准确性。ChatGPT-4与ChatGPT-3.5中位评分均为6(IQR 1),优于Google Bard的4.5(2)和OcularBERT的2(1)(所有p≤8.4×10⁻³)。作者认为生成式聊天机器人有潜力回答训练领域外的问题,但实际应用前仍需进一步验证研究。
要点
- ChatGPT-4与ChatGPT-3.5中位评分均为6(IQR 1),Google Bard为4.5(2),OcularBERT为2(1)。
- ChatGPT-4的83.3%和ChatGPT-3.5的86.7%回答被评为「良好」,Google Bard为50%,OcularBERT为10%。
- ChatGPT-4与ChatGPT-3.5无「差」评回答,Google Bard有6.7%「差」评,OcularBERT有20%。
核心结论
- ChatGPT-4与ChatGPT-3.5表现最优,其次为Google Bard和OcularBERT。
- 生成式聊天机器人有潜力回答其原始训练领域之外的专业问题。
- 在实际应用前仍需进一步验证研究。
研究设计与样本
- 横断面研究,评估四种聊天机器人:生成式模型ChatGPT-4、ChatGPT-3.5、Google Bard和检索式模型OcularBERT。
- 评估45个问题(15个年龄相关性黄斑变性、15个糖尿病视网膜病变、15个其他)的回答准确性。
- 由三名设盲的视网膜专科医生采用三点Likert量表(2=良好无错误,1=临界,0=差且有显著不准确)评分,汇总评分范围为0至6。
- 根据评分者多数共识,将回答分类为「良好」「临界」或「差」。
主要发现
- ChatGPT-4与ChatGPT-3.5中位评分(IQR)均为6(1),Google Bard为4.5(2),OcularBERT为2(1)(所有p≤8.4×10⁻³)。
- 共识评估中,ChatGPT-4的83.3%回答和ChatGPT-3.5的86.7%回答被评为「良好」,超过Google Bard的50%和OcularBERT的10%(所有p≤1.4×10⁻²)。
- ChatGPT-4与ChatGPT-3.5无「差」评回答,Google Bard产生6.7%「差」评,OcularBERT产生20%。
- 按问题类型,ChatGPT-4仅在年龄相关性黄斑变性问题上优于Google Bard,ChatGPT-3.5在糖尿病视网膜病变和其他问题上优于Google Bard。
关键数值与推荐
- 评分采用三点Likert量表:2=良好无错误,1=临界,0=差且有显著不准确,汇总评分范围0至6。
- 共识分类为「良好」「临界」或「差」。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND/AIMS: To compare the performance of generative versus retrieval-based chatbots in answering patient inquiries regarding age-related macular degeneration (AMD) and diabetic retinopathy (DR). METHODS: We evaluated four chatbots: generative models (ChatGPT-4, ChatGPT-3.5 and Google Bard) and a retrieval-based model (OcularBERT) in a cross-sectional study. Their response accuracy to 45 questions (15 AMD, 15 DR and 15 others) was evaluated and compared. Three masked retinal specialists graded the responses using a three-point Likert scale: either 2 (good, error-free), 1 (borderline) or 0 (poor with significant inaccuracies). The scores were aggregated, ranging from 0 to 6. Based on majority consensus among the graders, the responses were also classified as 'Good', 'Borderline' or 'Poor' quality. RESULTS: Overall, ChatGPT-4 and ChatGPT-3.5 outperformed the other chatbots, both achieving median scores (IQR) of 6 (1), compared with 4.5 (2) in Google Bard, and 2 (1) in OcularBERT (all p ≤8.4×10-3). Based on the consensus approach, 83.3% of ChatGPT-4's responses and 86.7% of ChatGPT-3.5's were rated as 'Good', surpassing Google Bard (50%) and OcularBERT (10%) (all p ≤1.4×10-2). ChatGPT-4 and ChatGPT-3.5 had no 'Poor' rated responses. Google Bard produced 6.7% Poor responses, and OcularBERT produced 20%. Across question types, ChatGPT-4 outperformed Google Bard only for AMD, and ChatGPT-3.5 outperformed Google Bard for DR and others. CONCLUSION: ChatGPT-4 and ChatGPT-3.5 demonstrated superior performance, followed by Google Bard and OcularBERT. Generative chatbots are potentially capable of answering domain-specific questions outside their original training. Further validation studies are still required prior to real-world implementation.
引用
- Cheong KX, Zhang C, Tan TE, et al. Comparing generative and retrieval-based chatbots in answering patient questions regarding age-related macular degeneration and diabetic retinopathy. Br J Ophthalmol. 2024. PMID: 38749531. doi:10.1136/bjo-2023-324533
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。