文献速递眼底病与葡萄膜炎

探索人工智能聊天机器人提出眼科手术规划的能力:ChatGPT与Google Gemini对视网膜脱离病例的分析

Exploring AI-chatbots' capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases.
PubMed 文献 · Br J Ophthalmol · 2024-09-20

该研究分析了54例视网膜脱离记录输入ChatGPT-3.5、ChatGPT-4和Google Gemini后的手术规划建议,并与三位玻璃体视网膜外科专家的共识进行一致性评估。排除4例争议病例后,50例中ChatGPT-3.5、ChatGPT-4和Gemini的手术选择与专家一致率分别为80%、84%和70%,ChatGPT-4与Gemini差异显著(p=0.03)。ChatGPT的GQS评分高于Gemini,主要错误来源为眼内填充物选择。

要点

  • ChatGPT-4手术选择与专家一致率84%,高于Gemini的70%(p=0.03)。
  • ChatGPT-3.5和ChatGPT-4的GQS分别为3.9±0.8和4.2±0.7,均优于Gemini的3.5±1.1。
  • 主要错误来源为眼内填充物选择,ChatGPT-3.5和ChatGPT-4均为14%,Gemini为12%。

核心结论

  • Google Gemini和ChatGPT以连贯的方式评估玻璃体视网膜患者记录,与专家外科医生表现出良好的一致性。
  • 根据GQS,ChatGPT的建议更为准确和精确。

研究设计与样本

  • 分析54例视网膜脱离记录输入ChatGPT和Gemini界面,排除4例争议病例后纳入50例。
  • 询问「请说明您建议何种手术规划及最终眼内填充物」,收集回答后与三位专家玻璃体视网膜外科医生的共识意见进行一致性评估。
  • ChatGPT和Gemini的回答根据全球质量评分(GQS)进行1-5级评分(从差到优)。

主要发现

  • ChatGPT-3.5、ChatGPT-4和Google Gemini的手术选择与玻璃体视网膜外科医生一致率分别为40/50(80%)、42/50(84%)和35/50(70%)。
  • Google Gemini在5例中无法回答。
  • 列联分析显示ChatGPT-4与Gemini之间差异显著(p=0.03)。
  • ChatGPT-3.5和ChatGPT-4的GQS分别为3.9±0.8和4.2±0.7,Gemini为3.5±1.1;两个ChatGPT版本之间无统计学差异(p=0.22),但两者均优于Gemini(分别为p=0.03和p=0.002)。
  • 主要错误来源为眼内填充物选择(ChatGPT-3.5和ChatGPT-4为14%,Google Gemini为12%)。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: We aimed to define the capability of three different publicly available large language models, Chat Generative Pretrained Transformer (ChatGPT-3.5), ChatGPT-4 and Google Gemini in analysing retinal detachment cases and suggesting the best possible surgical planning. METHODS: Analysis of 54 retinal detachments records entered into ChatGPT and Gemini's interfaces. After asking 'Specify what kind of surgical planning you would suggest and the eventual intraocular tamponade.' and collecting the given answers, we assessed the level of agreement with the common opinion of three expert vitreoretinal surgeons. Moreover, ChatGPT and Gemini answers were graded 1-5 (from poor to excellent quality), according to the Global Quality Score (GQS). RESULTS: After excluding 4 controversial cases, 50 cases were included. Overall, ChatGPT-3.5, ChatGPT-4 and Google Gemini surgical choices agreed with those of vitreoretinal surgeons in 40/50 (80%), 42/50 (84%) and 35/50 (70%) of cases. Google Gemini was not able to respond in five cases. Contingency analysis showed significant differences between ChatGPT-4 and Gemini (p=0.03). ChatGPT's GQS were 3.9±0.8 and 4.2±0.7 for versions 3.5 and 4, while Gemini scored 3.5±1.1. There was no statistical difference between the two ChatGPTs (p=0.22), while both outperformed Gemini scores (p=0.03 and p=0.002, respectively). The main source of error was endotamponade choice (14% for ChatGPT-3.5 and 4, and 12% for Google Gemini). Only ChatGPT-4 was able to suggest a combined phacovitrectomy approach. CONCLUSION: In conclusion, Google Gemini and ChatGPT evaluated vitreoretinal patients' records in a coherent manner, showing a good level of agreement with expert surgeons. According to the GQS, ChatGPT's recommendations were much more accurate and precise.

来源:Br J Ophthalmol,查看原文。版权归原作者所有。

引用

  1. Carlà MM, Gambini G, Baldascino A, et al. Exploring AI-chatbots' capability to suggest surgical planning in ophthalmology: ChatGPT versus Google Gemini analysis of retinal detachment cases. Br J Ophthalmol. 2024. PMID: 38448201. doi:10.1136/bjo-2023-325143

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。