ChatGPT能否为提示良性前列腺增大的男性下尿路症状提供高质量患者信息?
本研究以欧洲泌尿外科学会和美国泌尿外科学会的患者信息网站为参考,构建88个关于良性前列腺增大(BPE)相关下尿路症状(LUTS)的问题,评估ChatGPT 4.0+回答的准确性与质量。结果显示ChatGPT回答了全部88个问题,F1评分为0.79,精确率0.66,召回率0.97,质量评分中位数为4分,检查者间一致性良好(ICC=0.86)。作者认为ChatGPT在患者教育方面有潜在用途,但不应作为唯一信息来源,需进一步研究。
要点
- ChatGPT回答了全部88个BPE/LUTS相关问题,F1评分0.79,精确率0.66,召回率0.97。
- 回答质量评分中位数为4分(范围1-5),检查者间一致性良好(ICC=0.86)。
- 作者认为ChatGPT在患者教育方面有潜在用途,但不应作为唯一信息来源。
核心结论
- ChatGPT在向患者普及BPE/LUTS及其预后和治疗方面显示出潜在用途,有助于决策过程。
- 作者认为在推荐其作为唯一信息来源时应保持谨慎。
- 需要进一步研究以全面了解人工智能在泌尿外科患者教育中的效力范围。
研究设计与样本
- 研究以欧洲泌尿外科学会和美国泌尿外科学会的患者信息网站为参考材料,针对ChatGPT 4.0+构建了88个以BPE为中心的问题。
- 由检查者独立并双份比较ChatGPT的回答与参考材料,通过F1评分、精确率和召回率指标计算准确性。
- 采用5点李克特量表进行质量评分,使用组内相关系数评估检查者一致性,并使用Wilcoxon符号秩检验评估质量评分的差异。
主要发现
- ChatGPT回答了全部88个LUTS/BPE相关问题(88/88)。
- 在88个问题中,F1评分为0.79(范围0-1),精确率为0.66(范围0-1),召回率为0.97(范围0-1)。
- 质量评分中位数为4分(范围1-5)。
- 检查者间一致性良好(ICC=0.86)。
- 检查者给出的评分与回答总体质量之间无统计学显著差异(p=0.72)。
临床怎么用
- 作者认为ChatGPT在患者教育方面有潜在用途,但不应作为唯一信息来源。
- 作者指出需要进一步研究以全面了解人工智能在泌尿外科患者教育中的效力范围。
局限性
- 需要进一步研究以全面了解人工智能在泌尿外科患者教育中的效力范围。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: ChatGPT has recently emerged as a novel resource for patients' disease-specific inquiries. There is, however, limited evidence assessing the quality of the information. We evaluated the accuracy and quality of the ChatGPT's responses on male lower urinary tract symptoms (LUTS) suggestive of benign prostate enlargement (BPE) when compared to two reference resources. METHODS: Using patient information websites from the European Association of Urology and the American Urological Association as reference material, we formulated 88 BPE-centric questions for ChatGPT 4.0+. Independently and in duplicate, we compared the ChatGPT's responses and the reference material, calculating accuracy through F1 score, precision, and recall metrics. We used a 5-point Likert scale for quality rating. We evaluated examiner agreement using the interclass correlation coefficient and assessed the difference in the quality scores with the Wilcoxon signed-rank test. RESULTS: ChatGPT addressed all (88/88) LUTS/BPE-related questions. For the 88 questions, the recorded F1 score was 0.79 (range: 0-1), precision 0.66 (range: 0-1), recall 0.97 (range: 0-1), and the quality score had a median of 4 (range = 1-5). Examiners had a good level of agreement (ICC = 0.86). We found no statistically significant difference between the scores given by the examiners and the overall quality of the responses (p = 0.72). DISCUSSION: ChatGPT demostrated a potential utility in educating patients about BPE/LUTS, its prognosis, and treatment that helps in the decision-making process. One must exercise prudence when recommending this as the sole information outlet. Additional studies are needed to completely understand the full extent of AI's efficacy in delivering patient education in urology.
引用
- Puerto Nino AK, Garcia Perez V, Secco S, et al. Can ChatGPT provide high-quality patient information on male lower urinary tract symptoms suggestive of benign prostate enlargement? Prostate Cancer Prostatic Dis. 2024. PMID: 38871841. doi:10.1038/s41391-024-00847-7
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。