对ChatGPT生成的常见介入放射学操作患者教育手册的评估
本研究选取20项常见介入放射学操作,由5名用户用相同指令独立要求ChatGPT生成患者教育手册,再由两名独立放射科医生评估内容、质量与准确性。结果发现30%(30/100)的手册存在缺陷,共34处具体不准确,包括遗漏镇静信息(10/34)和操作相关并发症描述不准确(8/34);不同用户及不同操作间逐行比较存在统计学显著不一致(P < 0.001)。结论认为ChatGPT生成的手册具有潜在临床相关性和较一致的术语,但并非完全准确,存在缺陷及用户间结构差异,需进一步改进并保持人工监督与专家验证。
要点
- 30%(30/100)的ChatGPT生成患者教育手册存在缺陷,共34处具体不准确。
- 常见缺陷为遗漏操作镇静信息(10/34)和操作相关并发症描述不准确(8/34)。
- 不同用户及不同操作间手册逐行比较存在统计学显著不一致(P < 0.001)。
核心结论
- ChatGPT生成的患者教育手册具有潜在临床相关性和较一致的术语,但并非完全准确,存在缺陷及用户间结构差异,需进一步改进并保持人工监督与专家验证。
研究设计与样本
- 选取20项常见介入放射学操作,5名用户以相同指令独立要求ChatGPT生成患者教育手册,由两名独立放射科医生评估内容、质量与准确性。
诊断标准与关键数值
- 30%(30/100)的手册存在缺陷,共34处具体不准确。
- 34处不准确中,遗漏操作镇静信息10/34,操作相关并发症描述不准确8/34。
- 不同用户及不同操作间逐行比较存在统计学显著不一致(P < 0.001)。
临床应用与阅片要点
- 作者认为,为确保患者安全,需对大语言模型进一步改进和完善,同时保持人工监督与专家验证。
局限性
- 手册并非完全准确,存在缺陷及用户间结构差异。
- 作者指出需进一步改进和完善大语言模型,并保持人工监督与专家验证。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures.
METHODS AND MATERIALS: Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets.
RESULTS: In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P < 0.001).
CONCLUSION: ChatGPT-generated education pamphlets demonstrated potential clinical relevance and fairly consistent terminology; however, the pamphlets were not entirely accurate and exhibited some shortcomings and inter-user structural variabilities. To ensure patient safety, future improvements and refinements in large language models are warranted, while maintaining human supervision and expert validation.
引用
- Kooraki S, Hosseiny M, Jalili MH, et al. Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures. Acad Radiol. 2024. PMID: 38839458. doi:10.1016/j.acra.2024.05.024
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。