评估人工智能驱动的急性肝衰竭查询响应:准确性、清晰度和相关性的比较分析
该研究评估了ChatGPT 3.5、ChatGPT 4、BARD、CLAUDE和COPILOT五个大语言模型对16个急性肝衰竭问题的响应准确性、清晰度和相关性,并与增强检索生成技术增强的ChatGPT 4进行比较。结果显示ChatGPT 4结合增强检索生成在三个维度上得分最高(4.70、4.89、4.78),优于其他模型。
要点
- ChatGPT 4结合增强检索生成在准确性、清晰度和相关性上得分最高(4.70、4.89、4.78)。
- ChatGPT 4单独使用得分:准确性3.67、清晰度4.04、相关性4.01。
- BARD和COPILOT表现较低,准确性分别为2.01和2.26。
核心结论
- ChatGPT 4结合增强检索生成相比其他大语言模型表现更优,整合增强检索生成可提高响应清晰度、相关性和准确性。
- 人工智能模型必须持续演进并与医疗实践对齐,才能成功融入医疗保健。
研究设计与样本
- 基于真实世界临床使用和美国胃肠病学院指南制定16个急性肝衰竭问题或临床场景。
- 使用「新聊天」功能分别处理每个查询以减少偏倚。
- 采用GPT-4的增强检索生成功能整合外部来源作为参考。
- 由4名独立研究者按1至5分的李克特量表评估准确性、清晰度和相关性。
主要发现
- ChatGPT 4结合增强检索生成在全部三个维度上得分最高:准确性4.70、清晰度4.89、相关性4.78。
- ChatGPT 4单独使用得分:准确性3.67、清晰度4.04、相关性4.01。
- CLAUDE得分:清晰度3.04、相关性3.6、准确性3.65。
- BARD得分:准确性2.01、相关性3.03。
- COPILOT得分:准确性2.26、相关性3.12。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: Recent advancements in artificial intelligence (AI), particularly through the deployment of large language models (LLMs), have profoundly impacted healthcare. This study assesses 5 LLMs-ChatGPT 3.5, ChatGPT 4, BARD, CLAUDE, and COPILOT-on their response accuracy, clarity, and relevance to queries concerning acute liver failure (ALF). We subsequently compare these results with ChatGPT4 enhanced with retrieval augmented generation (RAG) technology. METHODS: Based on real-world clinical use and the American College of Gastroenterology guidelines, we formulated 16 ALF questions or clinical scenarios to explore LLMs' ability to handle different clinical questions. Using the "New Chat" functionality, each query was processed individually across the models to reduce any bias. Additionally, we employed the RAG functionality of GPT-4, which integrates external sources as references to ground the results. All responses were evaluated on a Likert scale from 1 to 5 for accuracy, clarity, and relevance by 4 independent investigators to ensure impartiality. RESULTS: ChatGPT 4, augmented with RAG, demonstrated superior performance compared with others, consistently scoring the highest (4.70, 4.89, 4.78) across all 3 domains. ChatGPT 4 exhibited notable proficiency, with scores of 3.67 in accuracy, 4.04 in clarity, and 4.01 in relevance. In contrast, CLAUDE achieved 3.04 in clarity, 3.6 in relevance, and 3.65 in accuracy. Meanwhile, BARD and COPILOT exhibited lower performance levels; BARD recorded scores of 2.01 in accuracy and 3.03 in relevance, while COPILOT obtained 2.26 in accuracy and 3.12 in relevance. DISCUSSION: The study highlights Chat GPT 4 +RAG's superior performance compared with other LLMs. By integrating RAG with LLMs, the system combines generative language skills with accurate, up-to-date information. This improves response clarity, relevance, and accuracy, making them more effective in healthcare. However, AI models must continually evolve and align with medical practices for successful healthcare integration.
引用
- Malik S, Frey LJ, Gutman J, et al. Evaluating Artificial Intelligence-Driven Responses to Acute Liver Failure Queries: A Comparative Analysis Across Accuracy, Clarity, and Relevance. Am J Gastroenterol. 2024. PMID: 39688962. doi:10.14309/ajg.0000000000003255
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。