基于Transformer的大语言模型在识别非结构化出院小结中药物不良事件关系中的有效性:新加坡研究
该研究在新加坡去标识化出院小结(n=1647)上验证9个经生物医学语料预训练的Transformer大语言模型,将其微调(n=5088)后分类药物-不良事件对是否具有因果关系,并在MIMIC-III(n=4418)上评估泛化性。结果显示LLM-BiLSTM架构较传统机器学习基准提升F1值,BioM-ELECTRA-Large-BiLSTM平均F1提升16.1%(从0.64至0.74);追加规则(如忽略不良事件为药物已知适应症的药-事件对)可进一步降低假阳性率,精确率最高提升5.6%(增量0.04)。结论认为Transformer大语言模型优于传统方法,但规则带来的额外改善提示模型在该因果检测任务上仍存在不完美。
要点
- Transformer大语言模型在非结构化出院小结中识别药物-不良事件因果关系优于传统机器学习。
- BioM-ELECTRA-Large-BiLSTM平均F1提升16.1%(0.64至0.74)。
- 追加规则(如排除不良事件为药物已知适应症)使精确率最高提升5.6%(增量0.04)。
核心结论
- 基于Transformer的大语言模型在识别非结构化出院小结中因果相关的药物-不良事件对方面优于传统机器学习方法。
- 追加规则可进一步降低假阳性率并提升精确率,但模型在该因果检测任务上仍存在一定不完美。
研究设计与样本
- 研究将9个经生物医学领域语料预训练的Transformer大语言模型在标注数据(n=5088)上微调,以分类药物-不良事件对为因果相关或不相关。
- 模型在新加坡去标识化医院出院小结的文本片段(n=1647)上验证,并在MIMIC-III数据库的标注片段(n=4418)上评估泛化性。
- 模型性能与同一数据上传统机器学习模型先前设定的基准进行比较。
主要发现
- 采用LLM-BiLSTM架构时,基于Transformer的大语言模型较先前基准提升F1分数,BioM-ELECTRA-Large-BiLSTM平均F1分数提升16.1%(从0.64升至0.74)。
- 在基于LLM的预测上追加规则(如当不良事件为药物已知适应症时忽略该药物-不良事件对)可进一步降低假阳性率,精确率提升最高达5.6%(增量0.04)。
临床怎么用
- 临床药学智能体在从出院小结提取药物-不良事件因果关系时,可优先考虑基于Transformer的大语言模型而非传统机器学习模型。
- 在模型输出后追加规则(如排除不良事件为药物已知适应症的药-事件对)有助于降低假阳性率并提升精确率。
局限性
- 追加规则后性能仍有改善,提示大语言模型在该因果 relation 检测任务上仍存在一定不完美。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: Transformer-based large language models (LLMs) have transformed the field of natural language processing and led to significant advancements in various text processing tasks. However, the applicability of these LLMs in identifying related drug-adverse event (AE) pairs within clinical context may be limited by the prevalent use of non-standard sentence structures and grammar. METHOD: Nine transformer-based LLMs pre-trained on biomedical domain corpora are fine-tuned on annotated data (n = 5088) to classify drug-AE pairs in unstructured discharge summaries as causally related or unrelated. These LLMs are then validated on text segments from deidentified hospital discharge summaries from Singapore (n = 1647). To assess generalisability, the models are validated on annotated segments (n = 4418) from the Medical Information Mart for Intensive Care (MIMIC-III) database. Performance of LLMs in identifying related drug-AE pairs is then compared against a prior benchmark set by traditional machine learning models on the same data. RESULTS: Using an LLM-Bidirectional long short-term memory (LLM-BiLSTM) architecture, transformer-based LLMs improve F1 score as compared to prior benchmark with BioM-ELECTRA-Large-BiLSTM showing an average F1 score improvement of 16.1% (increase from 0.64 to 0.74). Applying additional rules on the LLM-based predictions, like ignoring drug-AE pairs when the AE is a known indication of the drug, results in a further reduction in false positive rates with precision increases of up to 5.6% (0.04 increment). CONCLUSION: Transformer-based LLMs outperform traditional machine learning methods in identifying causally related drug-AE pairs embedded within unstructured discharge summaries. Nonetheless the improvement in performance with rules indicates that LLMs still possess some degree of imperfection for this causal relation detection task.
引用
- Koon YL, Lam YT, Tan HX, et al. Effectiveness of Transformer-Based Large Language Models in Identifying Adverse Drug Reaction Relations from Unstructured Discharge Summaries in Singapore. Drug Saf. 2025. PMID: 39982676. doi:10.1007/s40264-025-01525-w
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。