文献速递胃肠外科急腹症与外科感染围术期管理与 ERAS

从谷歌医生到ChatGPT医生:评估人工智能生成的阑尾炎医学信息的内容与质量

Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis.
PubMed 文献 · Surg Endosc · 2024-03-05

本研究评估了ChatGPT-3.5、ChatGPT-4、Bard和Claude-2四种生成式AI聊天机器人关于急性阑尾炎的医学信息质量与可读性。采用改良DISCERN工具(16项标准,每项1-5分,总分16-80分)由三名研究者盲法独立评分,并用Flesch阅读易读性(FRE)和Flesch-Kincaid年级水平(FKGL)评估可读性。结果显示AI生成内容质量评分较好,但多数未提供来源或提供虚构来源,且可读性远超公众推荐水平。

要点

  • ChatGPT-3.5、ChatGPT-4、Bard和Claude-2的总体平均质量评分分别为60.7、62.0、62.3和51.3(满分80分)。
  • Claude-2质量评分显著低于ChatGPT-4(p=0.001)、ChatGPT-3.5(p=0.005)和Bard(p=0.001)。
  • 仅Bard列出了可验证来源,Claude-2提供了虚构来源;除Claude-2外所有聊天机器人都建议出现症状时咨询医生。

核心结论

  • AI生成的阑尾炎医学信息在质量评估中得分较好,但多数未提供来源或提供虚构来源,且总体可读性远超公众推荐水平。
  • 生成式AI平台在阑尾炎患者教育和互动方面显示出可测量的潜力。

研究设计与样本

  • 评估AI生成医学信息内容与质量的研究,使用改良DISCERN评估工具(16项标准,每项5点Likert量表,总分范围16-80分)。
  • 四种常用聊天机器人ChatGPT-3.5、ChatGPT-4、Bard和Claude-2被提示生成关于阑尾炎的医学信息。
  • 三名研究者对生成的文本独立评分,且对AI平台身份盲法。
  • 可读性通过Flesch阅读易读性(FRE)和Flesch-Kincaid年级水平(FKGL)评分确定。

主要发现

  • ChatGPT-3.5、ChatGPT-4、Bard和Claude-2的总体平均(SD)质量评分分别为60.7(1.2)、62.0(1.0)、62.3(1.2)和51.3(2.3),评分量表为16-80分。
  • 评分者间信度分别为0.81、0.75、0.81和0.72,表明一致性较高。
  • Claude-2的平均质量评分显著低于ChatGPT-4(p=0.001)、ChatGPT-3.5(p=0.005)和Bard(p=0.001)。
  • Bard是唯一列出可验证来源的AI平台,而Claude-2提供了虚构来源。
  • 除Claude-2外,所有聊天机器人都建议读者出现症状时咨询医生。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: Generative artificial intelligence (AI) chatbots have recently been posited as potential sources of online medical information for patients making medical decisions. Existing online patient-oriented medical information has repeatedly been shown to be of variable quality and difficult readability. Therefore, we sought to evaluate the content and quality of AI-generated medical information on acute appendicitis. METHODS: A modified DISCERN assessment tool, comprising 16 distinct criteria each scored on a 5-point Likert scale (score range 16-80), was used to assess AI-generated content. Readability was determined using the Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) scores. Four popular chatbots, ChatGPT-3.5 and ChatGPT-4, Bard, and Claude-2, were prompted to generate medical information about appendicitis. Three investigators independently scored the generated texts blinded to the identity of the AI platforms. RESULTS: ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 had overall mean (SD) quality scores of 60.7 (1.2), 62.0 (1.0), 62.3 (1.2), and 51.3 (2.3), respectively, on a scale of 16-80. Inter-rater reliability was 0.81, 0.75, 0.81, and 0.72, respectively, indicating substantial agreement. Claude-2 demonstrated a significantly lower mean quality score compared to ChatGPT-4 (p = 0.001), ChatGPT-3.5 (p = 0.005), and Bard (p = 0.001). Bard was the only AI platform that listed verifiable sources, while Claude-2 provided fabricated sources. All chatbots except for Claude-2 advised readers to consult a physician if experiencing symptoms. Regarding readability, FKGL and FRE scores of ChatGPT-3.5, ChatGPT-4, Bard, and Claude-2 were 14.6 and 23.8, 11.9 and 33.9, 8.6 and 52.8, 11.0 and 36.6, respectively, indicating difficulty readability at a college reading skill level. CONCLUSION: AI-generated medical information on appendicitis scored favorably upon quality assessment, but most either fabricated sources or did not provide any altogether. Additionally, overall readability far exceeded recommended levels for the public. Generative AI platforms demonstrate measured potential for patient education and engagement about appendicitis.

来源:Surg Endosc,查看原文。版权归原作者所有。

引用

  1. Ghanem YK, Rouhi AD, Al-Houssan A, et al. Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis. Surg Endosc. 2024. PMID: 38443499. doi:10.1007/s00464-024-10739-5

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。