使用大型语言模型获取临床信息的希望与风险:ChatGPT作为生育咨询工具表现强劲但存在局限
本研究将2023年2月13日版ChatGPT与CDC不孕常见问题、两份生育知识量表及美国生殖医学会委员会意见进行对比,发现ChatGPT回答在篇幅、事实内容、情感极性和客观性上与CDC相似,但147条事实陈述中9条(6.12%)不正确,仅1条(0.68%)引用来源,且无法可靠引用来源和可能编造信息限制其临床使用。
要点
- ChatGPT回答CDC不孕FAQ的篇幅和事实内容与CDC来源相似,但147条事实陈述中9条(6.12%)不正确。
- ChatGPT仅1条(0.68%)陈述引用来源,无法可靠引用来源和可能编造信息限制其临床使用。
- ChatGPT在Cardiff生育知识量表上处于2013年国际队列第87百分位,在生育与不孕治疗知识评分上处于2017年队列第95百分位。
核心结论
- 2023年2月版ChatGPT能够对生育相关临床查询生成与已建立来源相当的相关、有意义的回答。
- 尽管通过医学领域特定训练可能改善性能,但无法可靠引用来源和不可预测的编造信息可能性可能限制其临床使用。
研究设计与样本
- 将2023年2月13日版ChatGPT与已建立的患者导向临床信息来源进行对比测试,包括CDC网站17个不孕常见问题、两份经验证的生育知识调查(Cardiff生育知识量表和生育与不孕治疗知识评分)以及美国生殖医学会委员会意见「优化自然生育」。
- 在2023年2月为期一周内,将常见问题、调查问题和改写后的总结陈述作为提示输入聊天机器人。
- 研究设置:学术医学中心;对象:在线AI聊天机器人。
主要发现
- 对于CDC的17个不孕FAQ,ChatGPT生成回答的篇幅相似(ChatGPT 207.8词/回答 vs CDC 181.0词/回答),事实内容相似(每回答8.65条事实陈述 vs 10.41条),情感极性相似(均值0.11 vs 0.11,量表-1负面至1正面),主观性相似(均值0.42 vs 0.35,量表0客观至1主观)。
- 147条ChatGPT事实陈述中9条(6.12%)被归类为不正确,仅1条(0.68%)陈述引用来源。
- ChatGPT在Cardiff生育知识量表上处于Bunting 2013年国际队列第87百分位,在生育与不孕治疗知识评分上基于Kudesia 2017年队列处于第95百分位。
- ChatGPT对「优化自然生育」中全部7条总结陈述复现了缺失事实。
局限性
- 无法可靠引用来源。
- 存在不可预测的编造信息可能性。
- 尽管通过医学领域特定训练可能改善性能,但上述局限可能限制其临床使用。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: To compare the responses of the large language model-based "ChatGPT" to reputable sources when given fertility-related clinical prompts. DESIGN: The "Feb 13" version of ChatGPT by OpenAI was tested against established sources relating to patient-oriented clinical information: 17 "frequently asked questions (FAQs)" about infertility on the Centers for Disease Control (CDC) Website, 2 validated fertility knowledge surveys, the Cardiff Fertility Knowledge Scale and the Fertility and Infertility Treatment Knowledge Score, as well as the American Society for Reproductive Medicine committee opinion "optimizing natural fertility." SETTING: Academic medical center. PATIENT(S): Online AI Chatbot. INTERVENTION(S): Frequently asked questions, survey questions and rephrased summary statements were entered as prompts in the chatbot over a 1-week period in February 2023. MAIN OUTCOME MEASURE(S): For FAQs from CDC: words/response, sentiment analysis polarity and objectivity, total factual statements, rate of statements that were incorrect, referenced a source, or noted the value of consulting providers. FOR FERTILITY KNOWLEDGE SURVEYS: Percentile according to published population data. FOR COMMITTEE OPINION: Whether response to conclusions rephrased as questions identified missing facts. RESULT(S): When administered the CDC's 17 infertility FAQ's, ChatGPT produced responses of similar length (207.8 ChatGPT vs. 181.0 CDC words/response), factual content (8.65 factual statements/response vs. 10.41), sentiment polarity (mean 0.11 vs. 0.11 on a scale of -1 (negative) to 1 (positive)), and subjectivity (mean 0.42 vs. 0.35 on a scale of 0 (objective) to 1 (subjective)). In total, 9 (6.12%) of 147 ChatGPT factual statements were categorized as incorrect, and only 1 (0.68%) statement cited a reference. ChatGPT would have been at the 87th percentile of Bunting's 2013 international cohort for the Cardiff Fertility Knowledge Scale and at the 95th percentile on the basis of Kudesia's 2017 cohort for the Fertility and Infertility Treatment Knowledge Score. ChatGPT reproduced the missing facts for all 7 summary statements from "optimizing natural fertility." CONCLUSION(S): A February 2023 version of "ChatGPT" demonstrates the ability of generative artificial intelligence to produce relevant, meaningful responses to fertility-related clinical queries comparable to established sources. Although performance may improve with medical domain-specific training, limitations such as the inability to reliably cite sources and the unpredictable possibility of fabricated information may limit its clinical use.
引用
- Chervenak J, Lieman H, Blanco-Breindel M, et al. The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertil Steril. 2023. PMID: 37217092. doi:10.1016/j.fertnstert.2023.05.151
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。