指南与共识技术与安全

利用GPT-4与上下文感知提升影像建议聊天机器人性能:提供可信赖的临床指导

Enhancing chatbot performance for imaging recommendations: Leveraging GPT-4 and context-awareness for trustworthy clinical guidance.
指南 / 共识 · Eur J Radiol · 2024-09-24

本研究将原有基于GPT-3.5-Turbo的上下文感知聊天机器人升级为GPT-4并改进提示策略,用美国放射学会 appropriateness criteria 文档提供个性化影像建议。升级版在提供「通常或可能适当」及「通常适当」建议方面优于原版、通用GPT-3.5-Turbo、通用GPT-4和普通放射科医生(均 p < 0.001),正确「通常适当」答案一致性78%,「通常或可能适当」一致性94%,且均选用相同源文档以保证透明。

要点

  • 上下文感知聊天机器人升级至GPT-4后,影像建议准确性优于通用GPT模型和普通放射科医生。
  • 一致性方面,78%的「通常适当」答案和94%的「通常或可能适当」答案正确且一致。
  • 所有情况下均选用相同源文档,输出可审计,增强临床决策支持透明度。

核心结论

  • 上下文感知对确保使用适当知识至关重要,升级后的聊天机器人可提高准确性、一致性和来源透明度,从而增强对聊天机器人输出的信任并改善临床决策支持过程。

研究设计与样本

  • 研究将原有基于GPT-3.5-Turbo的上下文感知聊天机器人升级为GPT-4并改进提示策略,与上一版本、通用GPT-3.5-Turbo、通用GPT-4及普通放射科医生比较应用ACR appropriateness指南的性能。

诊断标准与关键数值

  • 在提供「通常或可能适当」建议方面,升级版上下文感知聊天机器人优于上一版本、通用GPT-3.5-Turbo、通用GPT-4及普通放射科医生(均 p < 0.001)。
  • 在提供「通常适当」建议方面,升级版优于通用GPT-3.5-Turbo和普通放射科医生(均 p < 0.001)。
  • 正确「通常适当」答案的一致性为78%,「通常或可能适当」建议的一致性为94%。

临床应用与阅片要点

  • 升级版上下文感知聊天机器人可辅助放射科医生依据ACR指南提供个性化影像建议。
  • 所有情况下均选用相同源文档,保证输出透明可审计。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: To investigate if GPT-4 improves the accuracy, consistency, and trustworthiness of a context-aware chatbot to provide personalized imaging recommendations from American College of Radiology (ACR) appropriateness criteria documents using semantic similarity processing: In addition, we sought to enable auditability of the output by revealing the information source the decision relies on.

MATERIAL AND METHODS: We refined an existing chatbot that incorporated specialized knowledge of the ACR guidelines by upgrading GPT-3.5-Turbo to its successor GPT-4 by OpenAI, using the latest version of LlamaIndex, and improving the prompting strategy. This chatbot was compared to the previous version, generic GPT-3.5-Turbo and GPT-4, and general radiologists regarding the performance in applying the ACR appropriateness guidelines.

RESULTS: The refined context-aware chatbot performed superior to the previous version using GPT-3.5-Turbo, generic chatbots GPT-3.5-Turbo and GPT-4, and general radiologists in providing "usually or may be appropriate" recommendations according to the ACR guidelines (all p < 0.001). It also outperformed GPT-3.5-Turbo and general radiologists in respect to "usually appropriate" recommendations (both p < 0.001). Moreover, the consistency in correct answers was higher with 78 % consistent correct "usually appropriate" answers and 94 % for "usually or may be appropriate" recommendations. In all cases, the same source documents were chosen, ensuring transparency.

CONCLUSION: Our study demonstrates the significance of context awareness in ensuring the use of appropriate knowledge and proposes a strategy to enhance trust in chatbot-based outputs to provide transparency. The improvements in accuracy, consistency, and source transparency address trust issues and enhance the clinical decision support process.

ABBREVIATIONS: ACR, American College of Radiology; accGPT, appropriateness criteria context aware GPT; accGPT-4, appropriateness criteria context aware GPT using GPT-4; GPT, generative pre-trained transformer; LLM, Large Language Model.

来源:Eur J Radiol,查看原文。版权归原作者所有。

引用

  1. Rau A, Bamberg F, Fink A, et al. Enhancing chatbot performance for imaging recommendations: Leveraging GPT-4 and context-awareness for trustworthy clinical guidance. Eur J Radiol. 2024. PMID: 39326236. doi:10.1016/j.ejrad.2024.111756

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。