文献速递美容整形

人工智能用于患者支持:评估检索增强生成回答鼻整形术后问题的效果

Artificial Intelligence for Patient Support: Assessing Retrieval-Augmented Generation for Answering Postoperative Rhinoplasty Questions.
PubMed 文献 · Aesthet Surg J · 2025-01-01

本研究用30个常见患者问题测试四种检索增强生成(RAG)模型(Gemini-1.0-Pro-002、Gemini-1.5-Flash-001、Gemini-1.5-Pro-001和PaLM 2),评估其准确性、全面性、可读性和可理解性/可操作性。生成回答的完全准确率为41.7%,但无应答率达30.8%;Gemini-1.0-Pro-002全面性最优(P < .001),但可读性(FRE 40-49)和可理解性(均值0.7)低于患者教育标准。

要点

  • 生成回答的完全准确率为41.7%,无应答率为30.8%。
  • Gemini-1.0-Pro-002全面性最优(P < .001),PaLM 2可操作性最低(P < .007)。
  • 可读性(FRE 40-49)与可理解性(均值0.7)低于患者教育标准。

核心结论

  • 这是RAG首次应用于鼻整形术后患者护理,在准确性方面有优势,但也存在无应答和上下文理解等局限性。
  • 解决这些挑战将使RAG模型在多种外科和医疗场景中更安全、更有效地实施,有望通过减少医生工作量并增强患者参与来革新患者护理。

研究设计与样本

  • 研究测试四种RAG模型(Gemini-1.0-Pro-002、Gemini-1.5-Flash-001、Gemini-1.5-Pro-001和PaLM 2)对30个常见患者问题的回答。
  • 回答来源于权威鼻整形文本,评估准确性(1-5分)、全面性(1-3分)、可读性(Flesch阅读轻松度[FRE]、Flesch-Kincaid年级水平)和可理解性/可操作性(患者教育材料评估工具)。
  • 统计分析包括Wilcoxon秩和检验、Armitage趋势检验和成对比较。

主要发现

  • 生成回答时总体准确,完全准确率为41.7%;无应答率为30.8%,提示查询上下文解释和检索存在潜在挑战。
  • Gemini-1.0-Pro-002全面性最优(P < .001)。
  • 可读性(FRE:40-49)和可理解性(均值:0.7)低于患者教育标准。
  • PaLM 2可操作性得分最低(P < .007)。

关键数值与推荐

  • 准确性评估采用1-5分制,全面性评估采用1-3分制,可读性采用Flesch阅读轻松度和Flesch-Kincaid年级水平,可理解性/可操作性采用患者教育材料评估工具。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文报告的无应答率为30.8%,提示查询上下文解释和检索存在潜在挑战。
  • 可读性(FRE:40-49)和可理解性(均值:0.7)低于患者教育标准。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Although artificial intelligence (AI) is revolutionizing healthcare, inaccurate or incomplete information from pretrained large language models (LLMs) like ChatGPT poses significant risks to patient safety. Retrieval-augmented generation (RAG) offers a promising solution by leveraging curated knowledge bases to enhance accuracy and reliability, especially in high-demand specialties like plastic surgery. OBJECTIVES: This study evaluates the performance of RAG-enabled AI models in addressing postoperative rhinoplasty questions, aiming to assess their safety and identify necessary improvements for effective implementation into clinical care. METHODS: Four RAG models (Gemini-1.0-Pro-002, Gemini-1.5-Flash-001, Gemini-1.5-Pro-001, and PaLM 2) were tested on 30 common patient inquiries. Responses, sourced from authoritative rhinoplasty texts, were evaluated for accuracy (1-5 scale), comprehensiveness (1-3 scale), readability (Flesch Reading Ease [FRE], Flesch-Kincaid Grade Level), and understandability/actionability (Patient Education Materials Assessment Tool). Statistical analyses included Wilcoxon rank sum, Armitage trend tests, and pairwise comparisons. RESULTS: When responses were generated, they were generally accurate (41.7% completely accurate); however, a 30.8% nonresponse rate revealed potential challenges with query context interpretation and retrieval. Gemini-1.0-Pro-002 demonstrated superior comprehensiveness (P < .001), but readability (FRE: 40-49) and understandability (mean: 0.7) fell below patient education standards. PaLM 2 scored lowest in actionability (P < .007). CONCLUSIONS: This first application of RAG to postoperative rhinoplasty patient care highlights its strengths in accuracy alongside its limitations, including nonresponse and contextual understanding. Addressing these challenges will enable safer, more effective implementation of RAG models across diverse surgical and medical contexts, with the potential to revolutionize patient care by reducing physician workload while enhancing patient engagement.

来源:Aesthet Surg J,查看原文。版权归原作者所有。

引用

  1. Genovese A, Prabha S, Borna S, et al. Artificial Intelligence for Patient Support: Assessing Retrieval-Augmented Generation for Answering Postoperative Rhinoplasty Questions. Aesthet Surg J. 2025. PMID: 40088460. doi:10.1093/asj/sjaf038

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。