文献速递药物不良反应与药物警戒

自然语言机器学习模型用于安全性文献监测的验证

Validation of a Natural Language Machine Learning Model for Safety Literature Surveillance.
PubMed 文献 · Drug Saf · 2023-11-08

本研究前瞻性验证阿斯利康的文献监测分诊深度学习模型,与人工监测团队平行运行比较真实世界性能。模型召回率与现有监测团队相当,仅在严格证明非劣效的产品上获批使用。

要点

  • 深度学习文献分诊模型召回率与人工监测团队相当。
  • 模型仅在严格证明非劣效的产品上获批使用。
  • 前瞻性真实世界验证可全面考察模型一致性与失败模式。

核心结论

  • 深度学习文献监测分诊模型在真实世界条件下召回率与现有人工监测团队相当。
  • 模型被接受用于那些严格证明其相对于人工流程非劣效的产品。

研究设计与样本

  • 前瞻性验证研究,将文献监测分诊模型的真实世界性能与平行工作的人工监测团队进行比较。
  • 主要评价指标为模型召回率,以评估遗漏安全信号(模型假阴性)的风险。

主要发现

  • 模型从训练到测试表现出 consistent 的全局性能,召回率与现有监测团队相当。

临床怎么用

  • 模型可在严格证明非劣效的产品范围内用于文献监测分诊。
  • 前瞻性真实世界验证有助于考察模型一致性和失败模式,从而支持其用于监测流程。

局限性

  • 深度学习算法引入独特风险,其在良好药物警戒规范下的验证仍是未决问题。
  • 修改分诊流程的最大风险是遗漏安全信号,导致模型假阴性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: As part of routine safety surveillance, thousands of articles of potential interest are manually triaged for review by safety surveillance teams. This manual triage task is an interesting candidate for automation based on the abundance of process data available for training, the performance of natural language processing algorithms for this type of cognitive task, and the small number of safety signals that originate from literature review, resulting in its lower risk profile. However, deep learning algorithms introduce unique risks and the validation of such models for use in Good Pharmacovigilance Practice remains an open question. OBJECTIVE: Qualifying an automated, deep learning approach to literature surveillance for use at AstraZeneca. METHODS: The study is a prospective validation of a literature surveillance triage model, comparing its real-world performance with that of human surveillance teams working in parallel. The biggest risk in modifying this triage process is missing a safety signal (resulting in model false negatives) and hence model recall is the main evaluation metric considered. RESULTS: The model demonstrates consistent global performance from training through testing, with recall rates comparable to that of existing surveillance teams. The model is accepted for use specifically for those products where non-inferiority to the manual process is rigorously demonstrated. CONCLUSION: Characterizing model performance prospectively, under real-world conditions, allows us to thoroughly examine model consistency and failure modes, qualifying it for use in our surveillance processes. We also identify potential future improvements and recognize the opportunity for the community to collaborate on this shared task.

来源:Drug Saf,查看原文。版权归原作者所有。

引用

  1. Park J, Djelassi M, Chima D, et al. Validation of a Natural Language Machine Learning Model for Safety Literature Surveillance. Drug Saf. 2023. PMID: 37938539. doi:10.1007/s40264-023-01367-4

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。