使用大语言模型从临床笔记中算法识别治疗中出现的不良事件:炎症性肠病试点研究
该研究标注了928例炎症性肠病患者的928份门诊笔记,共703例严重不良事件,并比较八种语言模型识别药物-严重不良事件配对的表现。UCSF-BERT数值性能最高,准确率88-92%,宏F1 61-68%,较既往模型高5-10%,在识别用药后住院事件上显著更优(P<0.01)。
要点
- UCSF-BERT识别药物-严重不良事件配对的准确率为88-92%,宏F1为61-68%。
- UCSF-BERT在识别用药后住院事件上显著优于其他模型(P<0.01)。
- 该模型准确率较既往已发表模型高5-10%。
核心结论
- UCSF-BERT在从临床笔记中检测严重不良事件的任务上,数值准确率优于既往方法。
- 研究结果支持使用大语言模型增强药物警戒的潜在价值。
研究设计与样本
- 研究标注了928例炎症性肠病患者的928份门诊笔记,用于识别非甾体免疫抑制剂治疗后发生的严重不良事件相关住院。
- 笔记中共包含703例严重不良事件,最常见为预期疗效失败。
- 比较了八种候选语言模型,包括新适配的临床大语言模型UCSF-BERT。
主要发现
- UCSF-BERT在识别药物-严重不良事件配对时数值性能最高,准确率88-92%,宏F1 61-68%。
- UCSF-BERT准确率比既往已发表模型高5-10%。
- UCSF-BERT在识别用药后住院事件上显著更优(P<0.01)。
临床怎么用
- 大语言模型如UCSF-BERT有望用于从门诊临床笔记中自动识别严重不良事件,辅助药物警戒。
- 当前性能仍需在多中心数据和更新架构(如GPT)中进一步适配和评估。
局限性
- 研究为试点性质,需在多中心数据中验证。
- 未来需使用更新架构如生成式预训练变换器(GPT)改进方法并评估模型性能。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California - San Francisco (UCSF)-BERT, to identify serious AEs (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF-BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF-BERT was significantly superior at identifying hospitalization events emergent to medication use (P < 0.01). LLMs like UCSF-BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre-trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.
引用
- Silverman AL, Sushil M, Bhasuran B, et al. Algorithmic Identification of Treatment-Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease. Clin Pharmacol Ther. 2024. PMID: 38459719. doi:10.1002/cpt.3226
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。