零样本人工智能与抗凝指南推荐的一致性评估:用于椎管内麻醉出血风险判断
该研究基于美国区域麻醉与疼痛医学学会指南构建10个假设患者场景与26种抗凝方案的260种组合,评估GPT-3.5在椎管内麻醉出血风险评估中的准确性与可靠性,并探讨结构化提示引导的影响。结果显示基线表现略高于随机水平,使用详细提示与明确指南后性能显著提升(AUC 0.70,95%CI 0.64-0.77),不同药物类别间表现存在差异。
要点
- GPT-3.5基线评估椎管内出血风险仅略高于随机水平。
- 详细提示与明确指南可显著提升模型性能(AUC 0.70)。
- 不同抗凝药物类别的评估表现存在差异。
核心结论
- GPT-3.5在评估椎管内麻醉出血风险方面具有潜力,但依赖精确的提示词。
- LLM整合应谨慎推进,需考虑安全性与隐私问题及模型局限性。
研究设计与样本
- 基于美国区域麻醉与疼痛医学学会指南构建10个假设患者场景与26种抗凝方案,共260种独特组合。
- 为LLM设计5种提示,从最小引导到明确指令,并将模型回答与基于指南的真值表比较。
- 采用准确率与受试者工作特征曲线下面积(AUC)作为性能指标。
主要发现
- GPT-3.5基线性能略高于随机水平。
- 使用详细提示与明确指南后性能显著提升,AUC为0.70(95%CI 0.64-0.77)。
- 不同药物类别间的性能存在差异。
临床怎么用
- LLM可用于辅助椎管内麻醉出血风险的临床决策,但须依赖准确且相关的提示词。
- 整合LLM时应考虑安全与隐私问题,并谨慎对待其局限性。
局限性
- LLM性能依赖精确提示,基线表现仅略高于随机水平。
- 研究仅测试GPT-3.5,结果可能不推广至其他模型。
- 需进一步研究优化LLM性能并处理复杂场景。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: Artificial intelligence and large language models (LLMs) have emerged as potentially disruptive technologies in healthcare. In this study GPT-3.5, an accessible LLM, was assessed for its accuracy and reliability in performing guideline-based evaluation of neuraxial bleeding risk in hypothetical patients on anticoagulation medication. The study also explored the impact of structured prompt guidance on the LLM's performance. METHODS: A dataset of 10 hypothetical patient stems and 26 anticoagulation profiles (260 unique combinations) was developed based on American Society of Regional Anesthesia and Pain Medicine guidelines. Five prompts were created for the LLM, ranging from minimal guidance to explicit instructions. The model's responses were compared with a "truth table" based on the guidelines. Performance metrics, including accuracy and area under the receiver operating curve (AUC), were used. RESULTS: Baseline performance of GPT-3.5 was slightly above chance. With detailed prompts and explicit guidelines, performance improved significantly (AUC 0.70, 95% CI (0.64 to 0.77)). Performance varied among medication classes. DISCUSSION: LLMs show potential for assisting in clinical decision making but rely on accurate and relevant prompts. Integration of LLMs should consider safety and privacy concerns. Further research is needed to optimize LLM performance and address complex scenarios. The tested LLM demonstrates potential in assessing neuraxial bleeding risk but relies on precise prompts. LLM integration should be approached cautiously, considering limitations. Future research should focus on optimization and understanding LLM capabilities and limitations in healthcare.
引用
- Hurley NC, Gupta RK, Schroeder KM, et al. Danger, Danger, Gaston Labat! Does zero-shot artificial intelligence correlate with anticoagulation guidelines recommendations for neuraxial anesthesia? Reg Anesth Pain Med. 2024. PMID: 38253610. doi:10.1136/rapm-2023-104868
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。