Open AI 输出针对前列腺癌的信息质量与适当性
研究将欧洲泌尿外科学会(EAU)2023年前列腺癌指南中的195条推荐转化为问题,系统询问 Chat-GPT 的8月3日版本,并由两位专家泌尿外科医生独立按1至4分评分。结果显示50/195(26%)完全正确、51/195(26%)正确但不充分、47/195(24%)正确与误导信息混合、47/195(24%)完全错误;随访与生活质量章节回答最准确,诊断与治疗章节表现最差。
要点
- Chat-GPT 回答前列腺癌 EAU 指南问题时准确性较差,仅26%完全正确。
- 诊断与治疗章节表现最差,完全错误答案分别占19%和30%。
核心结论
- Chat-GPT 在回答基于 EAU 前列腺癌指南推荐的问题时准确性较差。
- 未来研究应在充分训练后评估其表现。
研究设计与样本
- 根据 EAU 2023 指南前列腺癌部分收集的推荐,共准备195个问题。
- 所有问题系统呈现给 Chat-GPT 的8月3日版本,由两位专家泌尿外科医生独立评分(1分完全正确、2分正确但不充分、3分正确与误导信息混合、4分完全错误)。
- 按章节和推荐等级进行了亚组分析。
主要发现
- 共评估195条推荐:50/195(26%)完全正确,51/195(26%)正确但不充分,47/195(24%)正确与误导信息混合,47/195(24%)完全错误。
- Open AI 在回答随访与生活质量问题时尤为准确。
- 诊断和治疗章节表现最差,完全错误答案分别占19%和30%。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Chat-GPT, a natural language processing (NLP) tool created by Open-AI, can potentially be used as a quick source for obtaining information related to prostate cancer. This study aims to analyze the quality and appropriateness of Chat-GPT's responses to inquiries related to prostate cancer compared to those of the European Urology Association's (EAU) 2023 prostate cancer guidelines. Overall, 195 questions were prepared according to the recommendations gathered in the prostate cancer section of the EAU 2023 Guideline. All questions were systematically presented to Chat-GPT's August 3 Version, and two expert urologists independently assessed and assigned scores ranging from 1 to 4 to each response (1: completely correct, 2: correct but inadequate, 3: a mix of correct and misleading information, and 4: completely incorrect). Sub-analysis per chapter and per grade of recommendation were performed. Overall, 195 recommendations were evaluated. Overall, 50/195 (26%) were completely correct, 51/195 (26%) correct but inadequate, 47/195 (24%) a mix of correct and misleading and 47/195 (24%) incorrect. When looking at different chapters Open AI was particularly accurate in answering questions on follow-up and QoL. Worst performance was recorded for the diagnosis and treatment chapters with respectively 19% and 30% of the answers completely incorrect. When looking at the strength of recommendation, no differences in terms of accuracy were recorded when comparing weak and strong recommendations (p > 0,05). Chat-GPT has a poor accuracy when answering questions on the PCa EAU guidelines recommendations. Future studies should assess its performance after adequate training.
引用
- Lombardo R, Gallo G, Stira J, et al. Quality of information and appropriateness of Open AI outputs for prostate cancer. Prostate Cancer Prostatic Dis. 2024. PMID: 38228809. doi:10.1038/s41391-024-00789-0
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。