文献速递妇科

让我们聊聊宫颈癌:评估ChatGPT对宫颈癌问题的回答准确性

Let's chat about cervical cancer: Assessing the accuracy of ChatGPT responses to cervical cancer questions.
PubMed 文献 · Gynecol Oncol · 2023-11-21

该研究用64个源自专业学会网站和作者临床经验的宫颈癌问题测试ChatGPT,由两名妇科肿瘤主治医师按四级评分。53.1%回答正确且全面,29.7%正确但不全面,15.6%部分错误,1.6%完全错误;预防和生存/生活质量正确率最高,诊断最差。

要点

  • ChatGPT对宫颈癌预防和生存/生活质量问题回答准确率最高,分别为91.7%和93.8%。
  • 治疗类问题正确率71.4%,诊断类问题正确率仅33.3%。
  • 作者认为在用作妇科医生工具或推荐为患者资源前,需医生参与进一步开发该模型。

核心结论

  • ChatGPT能准确回答宫颈癌预防、生存和生活质量问题,但在诊断和治疗方面准确性较低。
  • 在将其用作妇科医生工具或推荐为患者获取宫颈癌诊断和治疗信息的资源之前,应让医生参与该大型语言模型的进一步开发。

研究设计与样本

  • 研究使用64个改编自专业学会网站和作者临床经验的问题查询ChatGPT。
  • 由两名妇科肿瘤主治医师按四级量表评分:1)正确且全面,2)正确但不全面,3)部分正确部分错误,4)完全错误。
  • 评分差异由额外评审者解决。

主要发现

  • ChatGPT对34个(53.1%)问题给出正确且全面的回答。
  • 对19个(29.7%)问题给出正确但不全面的回答。
  • 对10个(15.6%)问题给出部分错误的回答。
  • 对1个(1.6%)问题给出完全错误的回答。
  • 预防类正确(评分1或2)比例为22/24(91.7%),生存/生活质量类为15/16(93.8%),治疗类为15/21(71.4%),诊断类仅1/3(33.3%)。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: To quantify the accuracy of ChatGPT in answering commonly asked questions pertaining to cervical cancer prevention, diagnosis, treatment, and survivorship/quality-of-life (QOL). METHODS: ChatGPT was queried with 64 questions adapted from professional society websites and the authors' clinical experiences. The answers were scored by two attending Gynecologic Oncologists according to the following scale: 1) correct and comprehensive, 2) correct but not comprehensive, 3) some correct, some incorrect, and 4) completely incorrect. Scoring discrepancies were resolved by additional reviewers as needed. The proportion of responses earning each score were calculated overall and within each question category. RESULTS: ChatGPT provided correct and comprehensive answers to 34 (53.1%) questions, correct but not comprehensive answers to 19 (29.7%) questions, partially incorrect answers to 10 (15.6%) questions, and completely incorrect answers to 1 (1.6%) question. Prevention and survivorship/QOL had the highest proportion of "correct" scores (scores of 1 or 2) at 22/24 (91.7%) and 15/16 (93.8%), respectively. ChatGPT performed less well in the treatment category, with 15/21 (71.4%) correct scores. It performed the worst in the diagnosis category with only 1/3 (33.3%) correct scores. CONCLUSION: ChatGPT accurately answers questions about cervical cancer prevention, survivorship, and QOL. It performs less accurately for cervical cancer diagnosis and treatment. Further development of this immensely popular large language model should include physician input before it can be utilized as a tool for Gynecologists or recommended as a patient resource for information on cervical cancer diagnosis and treatment.

来源:Gynecol Oncol,查看原文。版权归原作者所有。

引用

  1. Hermann CE, Patel JM, Boyd L, et al. Let's chat about cervical cancer: Assessing the accuracy of ChatGPT responses to cervical cancer questions. Gynecol Oncol. 2023. PMID: 37988948. doi:10.1016/j.ygyno.2023.11.008

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。