评估ChatGPT对甲状腺结节患者教育回答的质量
本研究以30个相同问题对ChatGPT进行4种提示条件下的120次查询,按错误、部分正确、正确、正确且有参考文献分级评分。总体69.2%回答至少正确,不同提示间至少部分正确与正确比例无差异,仅正确且有参考文献比例有差异;八年级水平提示平均阅读等级最低,但所有回答阅读等级均高于患者教育推荐水平,且存在AI幻觉。
要点
- ChatGPT对甲状腺结节多数问题可给出至少正确的回答,69.2%回答至少正确。
- 不同提示策略未改善至少部分正确与正确的比例,仅影响正确且有参考文献的比例。
- 八年级水平提示平均阅读等级最低(13.43±2.86),仍远高于患者教育推荐水平。
核心结论
- ChatGPT总体上对甲状腺结节大多数问题提供适当回答,与提示方式无关。
- 尽管采用针对性提示策略,ChatGPT生成的回答阅读等级仍远高于向患者呈现医学信息的可接受推荐水平。
- 显著比例的AI幻觉可能使临床医生目前无法推荐当前版本ChatGPT作为患者教育工具。
研究设计与样本
- 对ChatGPT进行4次查询,每次30个相同问题,共120个问题。
- 查询按初始提示分为无提示、患者友好提示、八年级水平提示和请求参考文献提示。
- 回答按错误、部分正确、正确、正确且有参考文献的分级评分。
- 采用卡方分析比较不同提示类型下各评分阈值回答的比例,采用方差分析评估提示类型与阅读等级的关系。
- 计算每个回答的Flesch-Kincaid阅读等级,并对ChatGPT回答中提供的参考文献进行汇总和真实性分析。
主要发现
- 所有提示共120个问题中,83个回答(69.2%)至少正确。
- 至少部分正确的回答比例在不同提示间无差异(p=0.795),正确的回答比例在不同提示间无差异(p=0.402)。
- 正确且有参考文献的回答比例在不同提示间有差异(p<0.0001)。
- 八年级水平提示的平均阅读等级最低(13.43±2.86),显著低于无提示(14.97±2.01,p=0.01)和请求参考文献提示(16.43±2.05,p<0.0001)。
- 请求参考文献提示在回答中生成80/80(100%)的参考文献医学出版物,其中70条(87.5%)为合法引用,58/80(72.5%)准确报告了所引用出版物的信息。
临床怎么用
- 原文结论指出,显著比例的AI幻觉可能使临床医生目前无法推荐当前版本ChatGPT作为患者教育工具。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Background: ChatGPT, an artificial intelligence (AI) chatbot, is the fastest growing consumer application in history. Given recent trends identifying increasing patient use of Internet sources for self-education, we seek to evaluate the quality of ChatGPT-generated responses for patient education on thyroid nodules. Methods: ChatGPT was queried 4 times with 30 identical questions. Queries differed by initial chatbot prompting: no prompting, patient-friendly prompting, 8th-grade level prompting, and prompting for references. Answers were scored on a hierarchical score: incorrect, partially correct, correct, or correct with references. Proportions of responses at incremental score thresholds were compared by prompt type using chi-squared analysis. Flesch-Kincaid grade level was calculated for each answer. The relationship between prompt type and grade level was assessed using analysis of variance. References provided within ChatGPT answers were totaled and analyzed for veracity. Results: Across all prompts (n = 120 questions), 83 answers (69.2%) were at least correct. Proportions of responses that were at least partially correct (p = 0.795) and correct (p = 0.402) did not differ by prompt; responses that were correct with references did (p < 0.0001). Responses from 8th-grade level prompting were the lowest mean grade level (13.43 ± 2.86) and were significantly lower than no prompting (14.97 ± 2.01, p = 0.01) and prompting for references (16.43 ± 2.05, p < 0.0001). Prompting for references generated 80/80 (100%) of referenced medical publications within answers. Seventy references (87.5%) were legitimate citations, and 58/80 (72.5%) provided accurately reported information from the referenced publication. Conclusion: ChatGPT overall provides appropriate answers to most questions on thyroid nodules regardless of prompting. Despite targeted prompting strategies, ChatGPT reliably generates responses corresponding to grade levels well-above accepted recommendations for presenting medical information to patients. Significant rates of AI hallucination may preclude clinicians from recommending the current version of ChatGPT as an educational tool for patients at this time.
引用
- Campbell DJ, Estephan LE, Sina EM, et al. Evaluating ChatGPT Responses on Thyroid Nodules for Patient Education. Thyroid. 2023. PMID: 38010917. doi:10.1089/thy.2023.0491
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。