文献速递药物不良反应与药物警戒

ADE Eval:药品说明书中不良事件提取文本处理系统评估

ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance.
PubMed 文献 · Drug Saf · 2020-10-02

美国FDA与MITRE组织共享任务ADE Eval,评估自然语言处理(NLP)算法从FDA批准药品说明书提取不良事件(ADE)的性能。13个团队提交23次运行,MedDRA编码最高F1为0.79,证据质量最高0.96,提及发现最高F1为0.89。结论认为NLP尚不能无干预使用,但值得探索将输出纳入人工药物警戒流程。

要点

  • NLP提取药品说明书ADE的MedDRA编码最高F1为0.79。
  • 证据质量最高分0.96,提及发现最高F1为0.89。
  • 当前NLP不宜完全替代人工,可辅助药物警戒流程。

核心结论

  • NLP技术尚未达到可无干预使用的水平,但值得探索将NLP输出纳入人工药物警戒工作流程。

研究设计与样本

  • FDA与MITRE组织的共享任务,评估多种NLP技术从公开FDA批准药品说明书识别ADE的性能。
  • 13个团队共提交23次运行,采用药物警戒专用注释指南和语料库。

主要发现

  • MedDRA编码最高F1为0.79,证据质量最高分为0.96,提及发现最高F1为0.89。

关键数值与推荐

  • MedDRA编码F1衡量算法为注释语料文本识别正确MedDRA术语的能力。
  • 证据质量分衡量算法正确识别支持所选MedDRA术语的注释文本比例。
  • 提及发现F1为校正系统输出用于后续训练的平均加权F1。

临床怎么用

  • 可将NLP输出作为辅助工具提供给药物警戒安全评估人员,但需人工审核干预。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: The US FDA is interested in a tool that would enable pharmacovigilance safety evaluators to automate the identification of adverse drug events (ADEs) mentioned in FDA prescribing information. The MITRE Corporation (MITRE) and the FDA organized a shared task-Adverse Drug Event Evaluation (ADE Eval)-to determine whether the performance of algorithms currently used for natural language processing (NLP) might be good enough for real-world use. OBJECTIVE: ADE Eval was conducted to evaluate a range of NLP techniques for identifying ADEs mentioned in publicly available FDA-approved drug labels (package inserts). It was designed specifically to reflect pharmacovigilance practices within the FDA and model possible pharmacovigilance use cases. METHODS: Pharmacovigilance-specific annotation guidelines and annotated corpora were created. Two metrics modeled the experiences of FDA safety evaluators: one measured the ability of an algorithm to identify correct Medical Dictionary for Regulatory Activities (MedDRA®) terms for the text from the annotated corpora, and the other assessed the quality of evidence extracted from the corpora to support the selected MedDRA® term by measuring the portion of annotated text an algorithm correctly identified. A third metric assessed the cost of correcting system output for subsequent training (averaged, weighted F1-measure for mention finding). RESULTS: In total, 13 teams submitted 23 runs: the top MedDRA® coding F1-measure was 0.79, the top quality score was 0.96, and the top mention-finding F1-measure was 0.89. CONCLUSION: While NLP techniques do not perform at levels that would allow them to be used without intervention, it is now worthwhile exploring making NLP outputs available in human pharmacovigilance workflows.

来源:Drug Saf,查看原文。版权归原作者所有。

引用

  1. Bayer S, Clark C, Dang O, et al. ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance. Drug Saf. 2020. PMID: 33006728. doi:10.1007/s40264-020-00996-3

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。