指南与共识妇科

ChatGPT 与卵巢癌管理国家指南的比较:ChatGPT 答对了吗——纪念斯隆凯特琳癌症中心卵巢癌团队研究

ChatGPT compared to national guidelines for management of ovarian cancer: Did ChatGPT get it right? - A Memorial Sloan Kettering Cancer Center Team Ovary study.
指南 / 共识 · Gynecol Oncol · 2024-07-22

该研究比较了 ChatGPT(GPT-4)与 NCCN 指南在卵巢癌管理中的表现。研究基于 NCCN 指南生成 10 个问题,由 5 名妇科肿瘤学家盲法评估准确性及完整性。结果显示,GPT-4 在风险因素、手术管理和监测方面准确且完整的比例高于 NCCN,但在药物治疗方面低于 NCCN。

要点

  • GPT-4 在风险因素、手术管理和监测问题上准确且完整的比例高于 NCCN 指南。
  • 在药物治疗问题上,GPT-4 准确且完整的比例低于 NCCN 指南。
  • GPT-4 偶有不准确,应限制聊天机器人的无监督使用。

核心结论

  • GPT-4 在某一时间点对有限的一组卵巢癌问题提供了准确且完整的回答,在风险因素、手术管理和监测方面表现最佳;但偶有不准确,应限制聊天机器人的无监督使用。

研究设计与样本

  • 基于 NCCN 指南生成 10 个关于卵巢癌管理的问题,由 5 名妇科肿瘤学家对 NCCN、无提示 GPT 和有提示 GPT 的回答进行盲法评估,评分 0 为不准确、1 为准确但不完整、2 为准确且完整。

主要发现

  • NCCN 回答中 48% 为准确且完整,无提示 GPT 为 64%,有提示 GPT 为 66%。
  • 准确但不完整的回答比例 NCCN 高于 GPT-4。
  • 风险因素、手术管理和监测问题的准确且完整评分比例 GPT-4 高于 NCCN,而药物治疗问题 GPT-4 低于 NCCN。
  • 无提示 GPT 回答中 14% 不准确,有提示 GPT 为 12%,NCCN 为 10%。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: We evaluated the performance of a chatbot compared to the National Comprehensive Cancer Network (NCCN) Guidelines for the management of ovarian cancer. METHODS: Using NCCN Guidelines, we generated 10 questions and answers regarding management of ovarian cancer at a single point in time. Questions were thematically divided into risk factors, surgical management, medical management, and surveillance. We asked ChatGPT (GPT-4) to provide responses without prompting (unprompted GPT) and with prompt engineering (prompted GPT). Responses were blinded and evaluated for accuracy and completeness by 5 gynecologic oncologists. A score of 0 was defined as inaccurate, 1 as accurate and incomplete, and 2 as accurate and complete. Evaluations were compared among NCCN, unprompted GPT, and prompted GPT answers. RESULTS: Overall, 48% of responses from NCCN, 64% from unprompted GPT, and 66% from prompted GPT were accurate and complete. The percentage of accurate but incomplete responses was higher for NCCN vs GPT-4. The percentage of accurate and complete scores for questions regarding risk factors, surgical management, and surveillance was higher for GPT-4 vs NCCN; however, for questions regarding medical management, the percentage was lower for GPT-4 vs NCCN. Overall, 14% of responses from unprompted GPT, 12% from prompted GPT, and 10% from NCCN were inaccurate. CONCLUSIONS: GPT-4 provided accurate and complete responses at a single point in time to a limited set of questions regarding ovarian cancer, with best performance in areas of risk factors, surgical management, and surveillance. Occasional inaccuracies, however, should limit unsupervised use of chatbots at this time.

来源:Gynecol Oncol,查看原文。版权归原作者所有。

引用

  1. Finch L, Broach V, Feinberg J, et al. ChatGPT compared to national guidelines for management of ovarian cancer: Did ChatGPT get it right? - A Memorial Sloan Kettering Cancer Center Team Ovary study. Gynecol Oncol. 2024. PMID: 39042956. doi:10.1016/j.ygyno.2024.07.007

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。