文献速递药物不良反应与药物警戒

Twitter不良事件识别系统的前瞻性评估:Web-RADR项目结果

Prospective Evaluation of Adverse Event Recognition Systems in Twitter: Results from the Web-RADR Project.
PubMed 文献 · Drug Saf · 2020-01-01

本研究开发了一个基于词典查找和逻辑回归分类器的Twitter不良事件识别系统,并在公开的WEB-RADR参考数据集上前瞻性评估其性能。系统在开发测试集上精确率0.53、召回率0.52、F1分数0.52,但在WEB-RADR数据集上降至精确率0.38、召回率0.20、F1分数0.26。研究识别出过拟合、选择偏倚、标签偏倚和患病率四个影响迁移性的因素,并建议使用独立基准数据集进行严格比较。

要点

  • 自动不良事件识别系统在独立数据集上性能显著下降,需警惕过拟合。
  • WEB-RADR参考数据集可作为评估系统迁移性的独立基准。
  • 影响迁移性的因素包括选择偏倚、标签偏倚和患病率。

核心结论

  • 自动不良事件识别系统在独立数据上的实际性能可能远低于已发表结果。
  • 实现通用的Twitter不良事件识别系统困难,可能解释实际药物警戒中缺乏此类系统。
  • 建议使用独立基准数据集评估迁移性并强制进行严格比较。

研究设计与样本

  • 开发基于词典查找和逻辑回归分类器的系统,使用196,533条手动标注AE关系的专有推文数据集。
  • 在公开的WEB-RADR参考数据集上前瞻性评估系统性能,探索影响迁移性的不同方面。

主要发现

  • 系统在开发测试集上精确率0.53、召回率0.52、F1分数0.52。
  • 系统在WEB-RADR参考数据集上精确率0.38、召回率0.20、F1分数0.26。
  • 先前发表的自动检测不良事件帖子方法在另一数据集上精确率0.5、召回率0.92、F1分数0.65,在WEB-RADR数据集上降至精确率0.37、召回率0.63、F1分数0.46。
  • 识别出四个导致迁移性差的潜在因素:过拟合、选择偏倚、标签偏倚和患病率。

临床怎么用

  • 在临床药物警戒中应用自动不良事件识别系统前,应使用独立基准数据集验证其性能。
  • 不应仅基于已发表性能指标预期系统在实际环境中的表现。

局限性

  • 证据来自单一研究,系统性能在独立数据集上显著下降,迁移性有限。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: A large number of studies on systems to detect and sometimes normalize adverse events (AEs) in social media have been published, but evidence of their practical utility is scarce. This raises the question of the transferability of such systems to new settings. OBJECTIVES: The aims of this study were to develop an AE recognition system, prospectively evaluate its performance on an external benchmark dataset and identify potential factors influencing the transferability of AE recognition systems. METHODS: A pipeline based on dictionary lookups and logistic regression classifiers was developed using a proprietary dataset of 196,533 Tweets manually annotated for AE relations and prospectively evaluated the system on the publicly available WEB-RADR reference dataset, exploring different aspects affecting transferability. RESULTS: Our system achieved 0.53 precision, 0.52 recall and 0.52 F1-score on the development test set; however, when applied to the WEB-RADR reference dataset, system performance dropped to 0.38 precision, 0.20 recall and 0.26 F1-score. Similarly, a previously published method aiming at automatically detecting adverse event posts reported 0.5 precision, 0.92 recall and 0.65 F1-score on thus another dataset, while performance on the WEB-RADR reference dataset was reduced to 0.37 precision, 0.63 recall and 0.46 F1-score. We identified four potential factors leading to poor transferability: overfitting, selection bias, label bias and prevalence. CONCLUSION: We warn the community about a potentially large discrepancy between the expected performance of automated AE recognition systems based on published results and the actual observed performance on independent data. This study highlights the difficulty of implementing an all-purpose system for automatic adverse event recognition in Twitter, which could explain the lack of such systems in practical pharmacovigilance settings. Our recommendation is to use benchmark independent datasets, such as the WEB-RADR reference, to investigate the transferability of the adverse event recognition systems and ultimately enforce rigorous comparisons across studies on the task.

来源:Drug Saf,查看原文。版权归原作者所有。

引用

  1. Gattepaille LM, Hedfors Vidlin S, Bergvall T, et al. Prospective Evaluation of Adverse Event Recognition Systems in Twitter: Results from the Web-RADR Project. Drug Saf. 2020. PMID: 32410156. doi:10.1007/s40264-020-00942-3

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。