生成式大语言模型在出院小结药物不良反应关系预测中的潜力与策略
该研究比较了生成式大语言模型在出院小结文本片段中预测药物与不良事件因果关系的能力。生成式大语言模型作为召回优化模型,F1分数与微调模型相当,Gemini 1.5 Pro和Llama 3.1 405B在MIMIC-Unrestricted数据集上优于内部微调的BioM-ELECTRA-Large。研究提出混合方法整合微调模型与生成式大语言模型,提升性能并节省计算资源。
要点
- 生成式大语言模型无需先期训练即可作为召回优化模型预测药物-不良事件关系。
- 混合方法在验证集上F1分数较BioM-ELECTRA-Large提升0.8%至18.5%。
- BioM-ELECTRA-Large仅选择19.7%至73.4%的片段供下游预测,节省计算资源。
核心结论
- 生成式大语言模型作为召回优化模型,其F1分数与微调模型相当,混合方法可提升性能并节省计算资源。
研究设计与样本
- 比较分析生成式大语言模型在出院小结文本片段中预测药物-不良事件因果关系的性能,使用MIMIC-Unrestricted等数据集。
主要发现
- 在MIMIC-Unrestricted数据集上,Gemini 1.5 Pro的F1分数提升19.2%(0.724-0.863),召回率提升39.7%(0.675-0.943)。
- Llama 3.1 405B的F1分数提升12.4%(0.724-0.814),召回率提升40.4%(0.675-0.948)。
- 混合模型在验证集上F1分数较BioM-ELECTRA-Large提升0.8%至18.5%(0.005-0.133),主要因精确率提高,但召回率较原始生成式大语言模型下降。
- BioM-ELECTRA-Large仅选择19.7%至73.4%的片段供下游生成式大语言模型预测,节省计算资源。
临床怎么用
- 可利用生成式大语言模型的高召回特性辅助识别出院小结中的药物不良反应关系,并结合微调模型提升精确率。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
We present a comparative analysis of generative large language models (LLMs) for predicting causal relationships between drugs and adverse events found in text segments from discharge summaries. Despite lacking prior training for identifying related drug-adverse event pairs, generative LLMs demonstrate exceptional performance as recall-optimized models, achieving F1 scores comparable to those of fine-tuned models. Notably, on the MIMIC-Unrestricted dataset, Gemini 1.5 Pro and Llama 3.1 405B outperform our in-house fine-tuned BioM-ELECTRA-Large, with Gemini 1.5 Pro showing a 19.2% (0.724-0.863) improvement in F1 score and a 39.7% (0.675-0.943) increase in recall, while Llama 3.1 405B exhibits a 12.4% (0.724-0.814) improvement in F1 and a 40.4% (0.675-0.948) boost in recall. Additionally, we propose a hybrid approach that integrates BioM-ELECTRA-Large with generative LLMs, resulting in enhanced performance over the individual models. Our hybrid model achieves F1 score improvements ranging from 0.8% to 18.5% (0.005-0.133) over BioM-ELECTRA-Large in the validation set, primarily due to increased precision, albeit with a decrease in recall compared with the original generative LLM. Importantly, this approach yields substantial computational resource savings, as BioM-ELECTRA-Large selects only a subset of segments-ranging from 19.7% to 73.4% across our datasets-for downstream prediction by generative LLMs. By harnessing the strengths of generative LLMs as recall-optimized models and combining them with fine-tuned models, we can unlock the full potential of artificial intelligence in predicting adverse drug reaction relations, ultimately enhancing patient safety.
引用
- Koon YL, Tan HX, Teo DCH, et al. Unlocking Potential of Generative Large Language Models for Adverse Drug Reaction Relation Prediction in Discharge Summaries: Analysis and Strategy. Clin Pharmacol Ther. 2025. PMID: 41114543. doi:10.1002/cpt.70100
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。