文献速递药物不良反应与药物警戒

开发用于药物警戒智能自动化的众包训练数据集

Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation.
PubMed 文献 · Drug Saf · 2020-12-22

本研究评估众包能否准确高效地生成药物警戒自动化所需的训练数据集。药物警戒专家基于15,490条去标识化社交媒体帖子建立参考数据集,并通过Amazon Turk众包标注相同医学概念。结果显示众包准确率超过90%,耗时约为专家的5%,提高报酬可提升时间效率但不影响准确率,多人员投票未显著改善准确率。

要点

  • 众包标注药物警戒训练数据准确率超过90%,耗时仅为专家的约5%。
  • 提高报酬可提升众包时间效率,但对准确率无显著影响。
  • 多名众包人员投票系统未显著提升标注准确率。

核心结论

  • 众包是准确且高效的方法,可用于开发支持药物警戒自动化的训练数据集。

研究设计与样本

  • 药物警戒专家通过审查15,490条去标识化社交媒体帖子(涉及15种药物和22个医学相关主题)创建参考数据集,并从参考数据集中随机抽样帖子发布在Amazon Turk上,由Turkers回答相同医学概念的问题,评估准确率、价格弹性和时间效率。

主要发现

  • 与参考数据集相比,众包 curation 的准确率超过90%,完成时间约为专家的5%。
  • 提高报酬可提升时间效率,但准确率无显著差异。
  • 由多名Turker使用投票系统审查社交媒体帖子未在准确率方面带来显著改善。

局限性

  • 需要更多研究以更好地理解可能用途的广度和深度以及优势、局限性和结果的普遍性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: Machine learning offers an alluring solution to developing automated approaches to the increasing individual case safety report burden being placed upon pharmacovigilance. Leveraging crowdsourcing to annotate unstructured data may provide accurate, efficient, and contemporaneous training data sets in support of machine learning. OBJECTIVE: The objective of this study was to evaluate whether crowdsourcing can be used to accurately and efficiently develop training data sets in support of pharmacovigilance automation. MATERIALS AND METHODS: Pharmacovigilance experts created a reference dataset by reviewing 15,490 de-identified social media posts of narratives pertaining to 15 drugs and 22 medically relevant topics. A random sampling of posts from the reference dataset was published on Amazon Turk and its users (Turkers) were asked a series of questions about those same medical concepts. Accuracy, price elasticity, and time efficiency were evaluated. RESULTS: Accuracy of crowdsourced curation exceeded 90% when compared to the reference dataset and was completed in about 5% of the time. There was an increase in time efficiency with higher pay, but there was no significant difference in accuracy. Additionally, having a social media post reviewed by more than one Turker (using a voting system) did not offer significant improvements in terms of accuracy. CONCLUSIONS: Crowdsourcing is an accurate and efficient method that can be used to develop training data sets in support of pharmacovigilance automation. More research is needed to better understand the breadth and depth of possible uses as well as strengths, limitations, and generalizability of results.

来源:Drug Saf,查看原文。版权归原作者所有。

引用

  1. Gartland A, Bate A, Painter JL, et al. Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation. Drug Saf. 2020. PMID: 33354751. doi:10.1007/s40264-020-01028-w

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。