评估人工智能驱动的患者教育:放射学案例研究
研究评估Bing聊天机器人(由ChatGPT驱动的大语言模型)对腹部CT、脊柱MRI和骨活检三类放射检查患者教育问题的回答准确性与完整性,共360次评估中93%评为「完全正确」、7%「大部分正确」,65%「完整」、35%「大部分完整」,可读性为八年级水平,无有害或误导性回答。
要点
- 360次评估中93%回答评为「完全正确」,7%为「大部分正确」,无错误或有害回答。
- 完整性方面65%评为「完整」、35%为「大部分完整」。
- 「更具创造性」设置产生更高比例的「完全正确」回答,其余设置或检查类型间无显著差异。
核心结论
- Bing聊天机器人能准确回答所问问题的全部或大部分方面,对细致问题倾向于谨慎作答,无错误或可能造成伤害或混淆的回答,大语言模型聊天机器人有潜力增强放射学患者教育并整合到患者门户中。
研究设计与样本
- 案例研究,纳入腹部CT、脊柱MRI和骨活检三种常见放射检查,每种检查10个问题,在三种聊天机器人设置下进行两轮试验,共360次评估,由两名评审独立对照radiologyinfo.org评估准确性和完整性。
诊断标准与关键数值
- 准确性:360次评估中336次(93%)为「完全正确」,24次(7%)为「大部分正确」;完整性:65%为「完整」,35%为「大部分完整」;可读性评级为八年级水平。
临床应用与阅片要点
- 作者认为大语言模型聊天机器人有潜力增强放射学患者教育,并可能整合到患者门户中用于检查准备和结果解读等多种目的。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: With recent advancements in the power and accessibility of artificial intelligence (AI) Large Language Models (LLMs) patients might increasingly turn to these platforms to answer questions regarding radiologic examinations and procedures, despite valid concerns about the accuracy of information provided. This study aimed to assess the accuracy and completeness of information provided by the Bing Chatbot-a LLM powered by ChatGPT-on patient education for common radiologic exams.
MATERIALS AND METHODS: We selected three common radiologic examinations and procedures: computed tomography (CT) abdomen, magnetic resonance imaging (MRI) spine, and bone biopsy. For each, ten questions were tested on the chatbot in two trials using three different chatbot settings. Two reviewers independently assessed the chatbot's responses for accuracy and completeness compared to an accepted online resource, radiologyinfo.org.
RESULTS: Of the 360 reviews performed, 336 (93%) were rated "entirely correct" and 24 (7%) were "mostly correct," indicating a high level of reliability. Completeness ratings showed that 65% were "complete" and 35% were "mostly complete." The "More Creative" chatbot setting produced a higher proportion of responses rated "entirely correct" but there were otherwise no significant difference in ratings based on chatbot settings or exam types. The readability level was rated eighth-grade level.
CONCLUSION: The Bing Chatbot provided accurate responses answering all or most aspects of the question asked of it, with responses tending to err on the side of caution for nuanced questions. Importantly, no responses were inaccurate or had potential to cause harm or confusion for the user. Thus, LLM chatbots demonstrate potential to enhance patient education in radiology and could be integrated into patient portals for various purposes, including exam preparation and results interpretation.
引用
- Kuckelman IJ, Yi PH, Bui M, et al. Assessing AI-Powered Patient Education: A Case Study in Radiology. Acad Radiol. 2023. PMID: 37709612. doi:10.1016/j.acra.2023.08.020
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。