人工智能聊天机器人在临床玻璃体视网膜场景中的表现评估
这项回顾性横断面研究设计了40个玻璃体视网膜疾病开放式临床场景,由三位盲法视网膜专科医生对ChatGPT回答的正确性和全面性评分。2023年6月ChatGPT正确回答83%(33/40)的场景,但仅52.5%(21/40)提供全面回答;错误回答中50%可能造成伤害、33.3%确定造成伤害。
要点
- ChatGPT正确回答了83%(33/40)的玻璃体视网膜临床场景。
- 仅52.5%(21/40)的场景获得全面回答。
- 错误回答中50%(3/6)可能造成伤害,33.3%(2/6)确定造成伤害。
核心结论
- ChatGPT正确回答了超过80%的复杂开放式玻璃体视网膜临床场景,但提供全面回答的能力有所降低。
研究设计与样本
- 回顾性横断面研究,设计40个涵盖玻璃体视网膜疾病四个主要主题的开放式临床场景,由三位盲法视网膜专科医生评分。
主要发现
- 2023年6月ChatGPT正确回答83%的临床场景(33/40),仅52.5%的场景(21/40)提供全面回答。
- 亚组分析显示平均正确评分:新生血管性年龄相关性黄斑变性86.7%,糖尿病视网膜病变100%,视网膜血管疾病76.7%,手术领域70%。
- 6例错误回答中,无伤害1例(16.7%),可能伤害3例(50%),确定伤害2例(33.3%)。
关键数值与推荐
- 主要结局为ChatGPT正确且全面回答的临床场景数。
- 次要结局包括对患者的理论伤害、聊天机器人所用参考文献类型的分布以及幻觉参考文献的频率。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
PURPOSE: To determine how often ChatGPT is able to provide accurate and comprehensive information regarding clinical vitreoretinal scenarios. To assess the types of sources ChatGPT primarily uses and to determine whether they are hallucinated. METHODS: This was a retrospective cross-sectional study. The authors designed 40 open-ended clinical scenarios across four main topics in vitreoretinal disease. Responses were graded on correctness and comprehensiveness by three blinded retina specialists. The primary outcome was the number of clinical scenarios that ChatGPT answered correctly and comprehensively. Secondary outcomes included theoretical harm to patients, the distribution of the type of references used by the chatbot, and the frequency of hallucinated references. RESULTS: In June 2023, ChatGPT answered 83% of clinical scenarios (33/40) correctly but provided a comprehensive answer in only 52.5% of cases (21/40). Subgroup analysis demonstrated an average correct score of 86.7% in neovascular age-related macular degeneration, 100% in diabetic retinopathy, 76.7% in retinal vascular disease, and 70% in the surgical domain. There were six incorrect responses with one case (16.7%) of no harm, three cases (50%) of possible harm, and two cases (33.3%) of definitive harm. CONCLUSION: ChatGPT correctly answered more than 80% of complex open-ended vitreoretinal clinical scenarios, with a reduced capability to provide a comprehensive response.
引用
- Maywood MJ, Parikh R, Deobhakta A, et al. PERFORMANCE ASSESSMENT OF AN ARTIFICIAL INTELLIGENCE CHATBOT IN CLINICAL VITREORETINAL SCENARIOS. Retina. 2024. PMID: 38271674. doi:10.1097/IAE.0000000000004053
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。