文献速递青光眼眼底病与葡萄膜炎

评估大型语言模型对青光眼和视网膜管理问题及病例的回答

Assessment of a Large Language Model's Responses to Questions and Cases About Glaucoma and Retina Management.
PubMed 文献 · JAMA Ophthalmol · 2024-01-01

这项横断面比较研究将GPT-4(2023年5月12日版本)对青光眼和视网膜问答及真实病例管理的回答,与15名青光眼和视网膜专科医生(12名主治医师、3名高年资住院医师)的回答进行Likert量表评分比较。结果显示,聊天机器人在准确性上优于青光眼专科医生,在完整性上与视网膜专科医生有差异,提示其作为眼科诊断辅助工具的潜力。

要点

  • 聊天机器人准确性平均秩506.2 vs 青光眼专科医生403.4(n=831,P<.001)。
  • 聊天机器人完整性平均秩528.3 vs 青光眼专科医生398.7(n=828,P<.001)。
  • 视网膜专科医生比较中,准确性平均秩235.3 vs 216.1(n=440,P=.17),完整性258.3 vs 208.7(n=439,P=.005)。

核心结论

  • 大型语言模型聊天机器人在诊断和治疗准确性上优于青光眼专科医生,与视网膜专科医生相当,支持其作为眼科诊断辅助工具有前景。

研究设计与样本

  • 比较性横断面研究,招募15名参与者(12名主治医师和3名高年资住院医师),年龄31至67岁,来自纽约西奈山伊坎医学院眼科附属诊所。
  • 从美国眼科学会“询问眼科医生”栏目随机选取青光眼和视网膜问题各10个,从西奈山附属诊所患者中随机选取去标识化的青光眼和视网膜病例各10个。
  • 使用的LLM为GPT-4(2023年5月12日版本),数据收集于2023年6月至8月。

主要发现

  • 问题与病例合并准确性平均秩:聊天机器人506.2 vs 青光眼专科医生403.4(n=831;Mann-Whitney U=27976.5;P<.001)。
  • 问题与病例合并完整性平均秩:聊天机器人528.3 vs 青光眼专科医生398.7(n=828;Mann-Whitney U=25218.5;P<.001)。
  • 视网膜专科医生比较准确性平均秩:聊天机器人235.3 vs 216.1(n=440;Mann-Whitney U=15518.0;P=.17)。
  • 视网膜专科医生比较完整性平均秩:聊天机器人258.3 vs 208.7(n=439;Mann-Whitney U=13123.5;P=.005)。
  • Dunn检验显示除专科医生与住院医师在评分聊天机器人完整性外,所有两两比较均有显著差异;专科医生评分聊天机器人准确性(z=3.23;P=.007)和完整性(z=5.86;P<.001)与住院医师相比有显著差异。

关键数值与推荐

  • 采用Likert量表评估回答的医学准确性和完整性。
  • 统计分析使用Mann-Whitney U检验和Kruskal-Wallis检验,随后进行两两比较。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

IMPORTANCE: Large language models (LLMs) are revolutionizing medical diagnosis and treatment, offering unprecedented accuracy and ease surpassing conventional search engines. Their integration into medical assistance programs will become pivotal for ophthalmologists as an adjunct for practicing evidence-based medicine. Therefore, the diagnostic and treatment accuracy of LLM-generated responses compared with fellowship-trained ophthalmologists can help assess their accuracy and validate their potential utility in ophthalmic subspecialties. OBJECTIVE: To compare the diagnostic accuracy and comprehensiveness of responses from an LLM chatbot with those of fellowship-trained glaucoma and retina specialists on ophthalmological questions and real patient case management. DESIGN, SETTING, AND PARTICIPANTS: This comparative cross-sectional study recruited 15 participants aged 31 to 67 years, including 12 attending physicians and 3 senior trainees, from eye clinics affiliated with the Department of Ophthalmology at Icahn School of Medicine at Mount Sinai, New York, New York. Glaucoma and retina questions (10 of each type) were randomly selected from the American Academy of Ophthalmology's commonly asked questions Ask an Ophthalmologist. Deidentified glaucoma and retinal cases (10 of each type) were randomly selected from ophthalmology patients seen at Icahn School of Medicine at Mount Sinai-affiliated clinics. The LLM used was GPT-4 (version dated May 12, 2023). Data were collected from June to August 2023. MAIN OUTCOMES AND MEASURES: Responses were assessed via a Likert scale for medical accuracy and completeness. Statistical analysis involved the Mann-Whitney U test and the Kruskal-Wallis test, followed by pairwise comparison. RESULTS: The combined question-case mean rank for accuracy was 506.2 for the LLM chatbot and 403.4 for glaucoma specialists (n = 831; Mann-Whitney U = 27976.5; P < .001), and the mean rank for completeness was 528.3 and 398.7, respectively (n = 828; Mann-Whitney U = 25218.5; P < .001). The mean rank for accuracy was 235.3 for the LLM chatbot and 216.1 for retina specialists (n = 440; Mann-Whitney U = 15518.0; P = .17), and the mean rank for completeness was 258.3 and 208.7, respectively (n = 439; Mann-Whitney U = 13123.5; P = .005). The Dunn test revealed a significant difference between all pairwise comparisons, except specialist vs trainee in rating chatbot completeness. The overall pairwise comparisons showed that both trainees and specialists rated the chatbot's accuracy and completeness more favorably than those of their specialist counterparts, with specialists noting a significant difference in the chatbot's accuracy (z = 3.23; P = .007) and completeness (z = 5.86; P < .001). CONCLUSIONS AND RELEVANCE: This study accentuates the comparative proficiency of LLM chatbots in diagnostic accuracy and completeness compared with fellowship-trained ophthalmologists in various clinical scenarios. The LLM chatbot outperformed glaucoma specialists and matched retina specialists in diagnostic and treatment accuracy, substantiating its role as a promising diagnostic adjunct in ophthalmology.

来源:JAMA Ophthalmol,查看原文。版权归原作者所有。

引用

  1. Huang AS, Hirabayashi K, Barna L, et al. Assessment of a Large Language Model's Responses to Questions and Cases About Glaucoma and Retina Management. JAMA Ophthalmol. 2024. PMID: 38386351. doi:10.1001/jamaophthalmol.2023.6917

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。