文献速递药物不良反应与药物警戒

利用机器学习从社交数字媒体中识别个体病例安全报告

Sorting Through the Safety Data Haystack: Using Machine Learning to Identify Individual Case Safety Reports in Social-Digital Media.
PubMed 文献 · Drug Saf · 2018-01-01

本研究开发了基于规则和机器学习的分类器,用于从社交数字媒体(SDM)帖子中识别个体病例安全报告(ICSR),并与人类药物警戒专家的一致性进行比较。最终模型在盲测集上达到gKappa 78%和准确率83%,处理311189条帖子仅需48小时,而人类专家估计需44000小时。结果表明自动化机器学习分类器可有效扩展ICSR检测工作流程。

要点

  • 机器学习分类器在盲测集上识别ICSR的gKappa为78%,准确率为83%。
  • 最终模型处理311189条帖子需48小时,人类专家估计需44000小时。
  • 自动化分类器可用于初筛ICSR,供人类专家进一步审查。

核心结论

  • 本研究表明,使用自动化机器学习分类器识别可能的ICSR以供人类专家进一步审查,是应对SDM中ICSR检测挑战的有效且可扩展的解决方案。

研究设计与样本

  • 研究从311189条提及罗氏产品和品牌的SDM帖子中随机抽样,帖子来源包括Twitter、Tumblr、Facebook和新闻媒体博客,开发并评估了三版自动化ICSR分类器。
  • 分类器模型包含注释相关ICSR元素的子组件和决定ICSR有效性的最终决策组件,以与人类药物警戒专家的一致性(Gwet AC1统计量,gKappa)作为首选性能指标。
  • 最佳性能模型在盲数据集上与罗氏全球药物警戒专家进行测试,并对全部311189条帖子数据集进行时间测试。

主要发现

  • 初始严格基于规则的ICSR分类模型准确率为65%,gKappa为46%。
  • 加入基于机器学习的不良事件注释器后,准确率提升至74%,gKappa提升至60%。
  • 进一步加入额外的机器学习ICSR检测器后,在2500条帖子的盲测集上,最终模型gKappa为78%,准确率为83%。
  • 在时间测试中,最终模型完成全部311189条帖子处理耗时48小时,而人类专家估计需44000小时。

临床怎么用

  • 可采用自动化机器学习分类器对社交数字媒体中的帖子进行初筛,识别可能的ICSR,再交由人类专家审查。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: There is increasing interest in social digital media (SDM) as a data source for pharmacovigilance activities; however, SDM is considered a low information content data source for safety data. Given that pharmacovigilance itself operates in a high-noise, lower-validity environment without objective 'gold standards' beyond process definitions, the introduction of large volumes of SDM into the pharmacovigilance workflow has the potential to exacerbate issues with limited manual resources to perform adverse event identification and processing. Recent advances in medical informatics have resulted in methods for developing programs which can assist human experts in the detection of valid individual case safety reports (ICSRs) within SDM. OBJECTIVE: In this study, we developed rule-based and machine learning (ML) models for classifying ICSRs from SDM and compared their performance with that of human pharmacovigilance experts. METHODS: We used a random sampling from a collection of 311,189 SDM posts that mentioned Roche products and brands in combination with common medical and scientific terms sourced from Twitter, Tumblr, Facebook, and a spectrum of news media blogs to develop and evaluate three iterations of an automated ICSR classifier. The ICSR classifier models consisted of sub-components to annotate the relevant ICSR elements and a component to make the final decision on the validity of the ICSR. Agreement with human pharmacovigilance experts was chosen as the preferred performance metric and was evaluated by calculating the Gwet AC1 statistic (gKappa). The best performing model was tested against the Roche global pharmacovigilance expert using a blind dataset and put through a time test of the full 311,189-post dataset. RESULTS: During this effort, the initial strict rule-based approach to ICSR classification resulted in a model with an accuracy of 65% and a gKappa of 46%. Adding an ML-based adverse event annotator improved the accuracy to 74% and gKappa to 60%. This was further improved by the addition of an additional ML ICSR detector. On a blind test set of 2500 posts, the final model demonstrated a gKappa of 78% and an accuracy of 83%. In the time test, it took the final model 48 h to complete a task that would have taken an estimated 44,000 h for human experts to perform. CONCLUSION: The results of this study indicate that an effective and scalable solution to the challenge of ICSR detection in SDM includes a workflow using an automated ML classifier to identify likely ICSRs for further human SME review.

来源:Drug Saf,查看原文。版权归原作者所有。

引用

  1. Comfort S, Perera S, Hudson Z, et al. Sorting Through the Safety Data Haystack: Using Machine Learning to Identify Individual Case Safety Reports in Social-Digital Media. Drug Saf. 2018. PMID: 29446035. doi:10.1007/s40264-018-0641-7

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。