大语言模型生成的肾结石患者信息材料的准确性与可读性:与官方泌尿外科组织材料的比较
本研究比较了大语言模型生成的肾结石患者信息材料与AUA、CUA、EAU官方材料的可读性和准确性。结果显示官方材料可读性优于聊天机器人输出,GPT-3.5可读性最差(13-14年级),GPT-4为11-13年级;官方材料准确,聊天机器人准确性高但遗漏次要细节;GPT-4在结石一般信息、饮食和内科管理方面比GPT-3.5更准确,两者在手术治疗方面准确性相同。
要点
- 官方泌尿外科组织的肾结石患者材料可读性优于GPT输出,但均高于普通人群阅读能力。
- GPT-3.5可读性最差(13-14年级),GPT-4为11-13年级,AUA材料最易读。
- 聊天机器人对常见肾结石问题准确性高但遗漏次要细节,临床医生需了解其优缺点。
核心结论
- 当前主要泌尿外科组织的肾结石患者信息材料仍比公开可用的GPT输出更易读,但仍高于普通人群的阅读能力。
- 在现有肾结石患者信息材料中,AUA的材料最易读。
- 聊天机器人对常见肾结石患者问题的输出准确性高但遗漏次要细节,临床医生了解其优势和局限性很重要。
研究设计与样本
- 研究获取并分类了AUA、CUA和EAU关于肾结石的患者信息材料。
- 从互联网查询中识别最常见的肾结石患者问题,并输入GPT-3.5和GPT-4。
- 使用先前发表的指数评估患者信息材料和ChatGPT输出的准确性和可读性。
- 评估了指定阅读水平(6年级)时ChatGPT输出的变化。
主要发现
- CUA材料可读性为10-12年级,AUA为8-10年级,EAU为9-11年级,均优于聊天机器人。
- GPT-3.5可读性最差,为13-14年级;GPT-4为11-13年级,同样比泌尿外科组织材料更难读。
- 组织材料被认为是准确的;聊天机器人准确性高但遗漏次要细节。
- GPT-4在结石一般信息、饮食和内科管理主题上比GPT-3.5更准确,两者在肾结石手术治疗主题上准确性相同。
局限性
- 原文未报告局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: To compare the readability and accuracy of large language model generated patient information materials (PIMs) to those supplied by the American Urological Association (AUA), Canadian Urological Association (CUA), and European Association of Urology (EAU) for kidney stones. METHODS: PIMs from AUA, CUA, and EAU related to nephrolithiasis were obtained and categorized. The most frequent patient questions related to kidney stones were identified from an internet query and input into GPT-3.5 and GPT-4. PIMs and ChatGPT outputs were assessed for accuracy and readability using previously published indexes. We also assessed changes in ChatGPT outputs when a reading level was specified (grade 6). RESULTS: Readability scores were better for PIMs from the CUA (grade level 10-12), AUA (8-10), or EAU (9-11) compared to the chatbot. GPT-3.5 had the worst readability scores at grade 13-14 and GPT-4 was likewise less readable than urologic organization PIMs with scores of 11-13. While organizational PIMs were deemed to be accurate, the chatbot had high accuracy with minor details omitted. GPT-4 was more accurate in general stone information, dietary and medical management of kidney stones topics in comparison to GPT-3.5, while both models had the same accuracy in the surgical management of nephrolithiasis topics. CONCLUSION: Current PIMs from major urologic organizations for kidney stones remain more readable than publicly available GPT outputs, but they are still higher than the reading ability of the general population. Of the available PIMs for kidney stones, those from the AUA are the most readable. Although Chatbot outputs for common kidney stone patient queries have a high degree of accuracy with minor omitted details, it is important for clinicians to understand their strengths and limitations.
引用
- Halawani A, Mitchell A, Saffarzadeh M, et al. Accuracy and Readability of Kidney Stone Patient Information Materials Generated by a Large Language Model Compared to Official Urologic Organizations. Urology. 2024. PMID: 38395071. doi:10.1016/j.urology.2023.11.042
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。