文献速递主动脉疾病颈动脉与弓上血管外周动脉疾病

生成式人工智能对血管外科患者回答的准确性与可读性:专家评估揭示安全风险与可及性差距

Accuracy and Readability of Generative Artificial Intelligence for Vascular Surgery Patients: A Specialist Based Evaluation Highlighting the Current Landscape of Safety Risks and Accessibility Gaps.
PubMed 文献 · Eur J Vasc Endovasc Surg · 2026-08-03

本研究由三位血管外科专科医生评估ChatGPT对16个血管外科常见问题的回答质量、可读性与临床适宜性。结果显示回答虽清晰但多不准确且需大学阅读水平,仅44%被全部评审认为临床适宜,症状相关问题的紧急度建议不一致构成安全隐患。

要点

  • ChatGPT血管外科回答清晰但仅44%被全部评审认为临床适宜。
  • 回答平均FRE 32.3、FKGL 13.5,需大学阅读水平,可读性不足。
  • 治疗相关问题仅25%适宜,症状相关问题紧急度建议不一致存安全隐患。

核心结论

  • ChatGPT血管外科回答清晰但多不临床适宜且不准确,需13年级阅读技能。
  • 通用大语言模型目前不能提供面向患者的血管外科信息。
  • 基于精选血管指南严格验证的领域特定知识锁定AI系统可能更合适。

研究设计与样本

  • 开发16个通俗风格问题,覆盖腹主动脉瘤、颈动脉狭窄、外周动脉闭塞病和静脉曲张。
  • 问题分属症状体征、自然病史、医疗建议和最佳治疗四领域,并分为症状相关与治疗相关组。
  • 向ChatGPT(gpt-3.5-turbo-0125)在专用会话中提问,采用改编QUEST、DISCERN和紧急度量表评估语气、互补性、紧急度和不确定性。
  • 用Flesch Reading Ease(FRE)和Flesch-Kincaid Grade Level(FKGL)测可读性。
  • 三位委员会认证血管外科医生用Likert量表评准确性、全面性和清晰度。

主要发现

  • 语气或互补性在不同疾病类别或问题类型间无显著差异。
  • 症状相关与治疗相关问题总体紧急度无显著差异,但症状相关子类型间紧急度差异显著(p = .030),与不一致的升级建议一致。
  • 平均FRE为32.3 ± 12.1,FKGL为13.5 ± 2,对应大学水平阅读要求。
  • 症状相关回答比治疗相关更易读(FRE 38.1 ± 10.3 vs. 26.4 ± 11.4;p = .025)。
  • 16个输出中仅7个(44%)被全部评审认为临床适宜;清晰度81%足够,准确性仅50%可接受,全面性69%可接受;治疗相关回答仅25%适宜。

关键数值与推荐

  • 平均FRE 32.3 ± 12.1,FKGL 13.5 ± 2对应大学水平阅读要求。
  • 症状相关回答FRE 38.1 ± 10.3 vs. 治疗相关26.4 ± 11.4(p = .025)。
  • 症状相关子类型间紧急度差异显著(p = .030)。
  • 16个输出中仅7个(44%)被全部评审认为临床适宜。
  • 清晰度81%足够,准确性50%可接受,全面性69%可接受,治疗相关仅25%适宜。

临床怎么用

  • 临床不宜直接使用通用大语言模型向血管外科患者提供信息。
  • 需关注症状相关问题中紧急度建议不一致的潜在安全风险。
  • 可考虑基于精选血管指南严格验证的领域特定知识锁定AI系统。

局限性

  • 仅评估ChatGPT(gpt-3.5-turbo-0125)单一模型。
  • 仅16个问题,样本量有限。
  • 由三位血管外科医生评审,存在主观性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: A study to evaluate the quality, readability, and clinical appropriateness of ChatGPT generated answers to common vascular surgery questions was designed. METHODS: Sixteen questions, written in a layperson style, were developed covering four major vascular conditions: abdominal aortic aneurysm; carotid stenosis; peripheral arterial occlusive disease; and varicose veins. The questions were categorised across four domains (signs and symptoms, natural history, medical advice, and best treatment) and further grouped for analysis into symptom related and treatment related queries. They were addressed to ChatGPT (gpt-3.5-turbo-0125) in dedicated sessions. Tone, complementarity, urgency, and uncertainty were assessed using adapted QUEST, DISCERN, and urgency scales. The Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) were used to measure readability. The accuracy, comprehensiveness, and clarity of the information was rated on a Likert scale by three board certified vascular surgeons. RESULTS: No significant differences were found in tone or complementarity across disease categories or question types. Urgency did not differ significantly between symptom related and treatment related queries overall; however, urgency varied significantly across sub-types of symptom related questions (p = .030), consistent with inconsistent escalation recommendations. The mean FRE was 32.3 ± 12.1 and FKGL 13.5 ± 2, corresponding to a university level reading requirement. Symptom related responses were more readable than treatment related ones (FRE 38.1 ± 10.3 vs. 26.4 ± 11.4; p = .025). Among the 16 outputs, only seven (44%) were judged clinically appropriate by all reviewers; clarity was rated adequate in 81% of responses, whereas only 50% reached acceptable accuracy and 69% acceptable comprehensiveness. Treatment related answers were particularly weak, with only 25% deemed appropriate. In symptom related questions, misalignment of urgency recommendations emerged as a potential patient safety concern. CONCLUSION: In this specialist evaluation, ChatGPT outputs related to vascular surgery were clear, but mostly clinically inappropriate and inaccurate, and required 13th grade reading skills. As they stand, generalist large language models cannot provide patient facing information about vascular surgery. A rigorously validated, domain specific, and knowledge locked artificial intelligence system based on curated vascular guidelines may be more appropriate to ensure safety and comprehension.

来源:Eur J Vasc Endovasc Surg,查看原文。版权归原作者所有。

引用

  1. D'Oria M, Dorigo W, Alexiou VG, et al. Accuracy and Readability of Generative Artificial Intelligence for Vascular Surgery Patients: A Specialist Based Evaluation Highlighting the Current Landscape of Safety Risks and Accessibility Gaps. Eur J Vasc Endovasc Surg. 2026. PMID: 42546836. doi:10.1016/j.ejvs.2026.07.060

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。