文献速递介入技术与安全

ChatGPT 对介入放射学操作潜在并发症的回答准确性及可读性:AI 辅助的患者访谈

Accuracy and Readability of ChatGPT on Potential Complications of Interventional Radiology Procedures: AI-Powered Patient Interviewing.
PubMed 文献 · Acad Radiol · 2024-11-16

研究让 ChatGPT-4 回答 25 种介入放射学操作(8 种非血管、17 种血管)的主要和次要并发症,由两位资深介入放射科医生依据欧洲心血管与介入放射学会指南用 5 分 Likert 量表评分,并用 Flesch-Kincaid 年级水平、Flesch 阅读易度及 SMOG 指数评估可读性;总分为 125 分,两位评分者分别给出 104 分和 109 分,差异无统计学意义(p=0.244),评分者间一致性高(ICC=0.928),可读性整体偏低,非血管与血管操作间可读性差异无统计学意义(p=0.16)。

要点

  • 两位介入放射科医生对 ChatGPT-4 回答的总评分分别为 104 分和 109 分(满分 125 分),差异无统计学意义(p=0.244)。
  • 两位评分者间一致性高(ICC=0.928),非血管与血管操作的 Pearson 相关系数分别为 0.908 和 0.896。
  • 可读性指标均处于大学水平:Flesch-Kincaid 年级水平 12.51±1.14、Flesch 阅读易度 30.27±8.38、SMOG 指数 14.46±0.76。

核心结论

  • ChatGPT-4 表现良好,可能有助于提高介入放射学操作信息的可及性并支持患者教育材料制作;但充分理解其回答需要较高的教育水平和健康素养。

研究设计与样本

  • 研究向 ChatGPT-4 询问 25 种介入放射学操作(8 种非血管、17 种血管)的潜在主要和次要并发症,由两位经验分别为 >25 年和 10 年的介入放射科医生依据欧洲心血管与介入放射学会指南采用 5 点 Likert 量表评分,并用 Wilcoxon 符号秩检验、组内相关系数(ICC)和 Pearson 相关系数评估评分一致性,同时用 Flesch-Kincaid 年级水平、Flesch 阅读易度评分和 SMOG 指数定量评估可读性与复杂度。

诊断标准与关键数值

  • 两位介入放射科医生对全部操作的总评分分别为 104 分和 109 分(满分 125 分),差异无统计学意义(p=0.244);8 种非血管操作两位均评 34 分(满分 40 分);17 种血管操作介入放射科医生 1 评 70 分、介入放射科医生 2 评 75 分(满分 85 分);评分者间 ICC=0.928,非血管和血管操作的 Pearson 相关系数分别为 0.908 和 0.896;平均 Flesch-Kincaid 年级水平 12.51±1.14、Flesch 阅读易度 30.27±8.38、SMOG 指数 14.46±0.76,均为大学水平;非血管与血管操作间可读性差异无统计学意义(p=0.16)。

临床应用与阅片要点

  • 作者认为 ChatGPT-4 表现良好,可能有助于提高介入放射学操作信息的可及性并支持患者教育材料制作;作者同时强调,充分理解其回答需要较高的教育水平和健康素养。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: It is crucial to inform the patient about potential complications and obtain consent before interventional radiology procedures. In this study, we investigated the accuracy, reliability, and readability of the information provided by ChatGPT-4 about potential complications of interventional radiology procedures.

MATERIALS AND METHODS: Potential major and minor complications of 25 different interventional radiology procedures (8 non-vascular, 17 vascular) were asked to ChatGPT-4 chatbot. The responses were evaluated by two experienced interventional radiologists (>25 years and 10 years of experience) using a 5-point Likert scale according to Cardiovascular and Interventional Radiological Society of Europe guidelines. The correlation between the two interventional radiologists' scoring was evaluated by the Wilcoxon signed-rank test, Intraclass Correlation Coefficient (ICC), and Pearson correlation coefficient (PCC). In addition, readability and complexity were quantitatively assessed using the Flesch-Kincaid Grade Level, Flesch Reading Ease scores, and Simple Measure of Gobbledygook (SMOG) index.

RESULTS: Interventional radiologist 1 (IR1) and interventional radiologist 2 (IR2) gave 104 and 109 points, respectively, out of a potential 125 points for the total of all procedures. There was no statistically significant difference between the total scores of the two IRs (p = 0.244). The IRs demonstrated high agreement across all procedure ratings (ICC=0.928). Both IRs scored 34 out of 40 points for the eight non-vascular procedures. 17 vascular procedures received 70 points out of 85 from IR1 and 75 from IR2. The agreement between the two observers' assessments was good, with PCC values of 0.908 and 0.896 for non-vascular and vascular procedures, respectively. Readability levels were overall low. The mean Flesch-Kincaid Grade Level, Flesch Reading Ease scores, and SMOG index were 12.51 ± 1.14 (college level) 30.27 ± 8.38 (college level), and 14.46 ± 0.76 (college level), respectively. There was no statistically significant difference in readability between non-vascular and vascular procedures (p = 0.16).

CONCLUSION: ChatGPT-4 demonstrated remarkable performance, highlighting its potential to enhance accessibility to information about interventional radiology procedures and support the creation of educational materials for patients. Based on the findings of our study, while ChatGPT provides accurate information and shows no evidence of hallucinations, it is important to emphasize that a high level of education and health literacy are required to fully comprehend its responses.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Kaba E, Beyazal M, Çeliker FB, et al. Accuracy and Readability of ChatGPT on Potential Complications of Interventional Radiology Procedures: AI-Powered Patient Interviewing. Acad Radiol. 2024. PMID: 39551684. doi:10.1016/j.acra.2024.10.028

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。