文献速递药物不良反应与药物警戒

药物警戒命名实体识别语料库:语言语域与训练数据充分性评估

Provision and Characterization of a Corpus for Pharmaceutical, Biomedical Named Entity Recognition for Pharmacovigilance: Evaluation of Language Registers and Training Data Sufficiency.
PubMed 文献 · Drug Saf · 2023-06-20

研究构建了一个包含18种实体、1400条记录(科学语言790条、非正式语言610条)的药物警戒命名实体识别数据集,并引入分数分层k折交叉验证评估实体层面训练数据充分性。结果表明,整合多语言语域的模型比单一语域模型性能更优或相当,且具有更好的可维护性和鲁棒性。

要点

  • 整合多语言语域的模型优于单一语域模型,性能相当或更高。
  • 分数分层k折交叉验证可评估实体层面训练数据充分性。
  • 数据集含1400条记录、2622句、9989个实体出现,已公开。

核心结论

  • 整合不同语言语域的模型比单一语域模型具有更好的可维护性、鲁棒性及相似或更高的性能。
  • 分数分层k折交叉验证可用于评估实体层面的训练数据充分性。

研究设计与样本

  • 构建了包含18种不同实体的数据集,合并科学语言与非正式语言两种语域。
  • 数据集共1400条记录(科学语言790条,非正式语言610条),2622个句子,9989个实体出现,来自外部801条和内部599条记录。
  • 采用分数分层k折交叉验证,以训练数据分数评估实体层面模型性能。

主要发现

  • 单一语言语域模型相比整合多语域训练的模型表现不佳。
  • 整合模型在实体层面性能相似或更高,且可维护性和鲁棒性更好。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION AND OBJECTIVE: Machine learning (ML) systems are widely used for automatic entity recognition in pharmacovigilance. Publicly available datasets do not allow the use of annotated entities independently, focusing on small entity subsets or on single language registers (informal or scientific language). The objective of the current study was to create a dataset that enables independent usage of entities, explores the performance of predictive ML models on different registers, and introduces a method to investigate entity cut-off performance. METHODS: A dataset has been created combining different registers with 18 different entities. We applied this dataset to compare the performance of integrated models with models created with single language registers only. We introduced fractional stratified k-fold cross-validation to determine model performance on entity level by using training dataset fractions. We investigated the course of entity performance with fractions of training datasets and evaluated entity peak and cut-off performance. RESULTS: The dataset combines 1400 records (scientific language: 790; informal language: 610) with 2622 sentences and 9989 entity occurrences and combines data from external (801 records) and internal sources (599 records). We demonstrated that single language register models underperform compared to integrated models trained with multiple language registers. CONCLUSIONS: A manually annotated dataset with a variety of different pharmaceutical and biomedical entities was created and is made available to the research community. Our results show that models that combine different registers provide better maintainability, have higher robustness, and have similar or higher performance. Fractional stratified k-fold cross-validation allows the evaluation of training data sufficiency on the entity level.

来源:Drug Saf,查看原文。版权归原作者所有。

引用

  1. Dietrich J, Kazzer P. Provision and Characterization of a Corpus for Pharmaceutical, Biomedical Named Entity Recognition for Pharmacovigilance: Evaluation of Language Registers and Training Data Sufficiency. Drug Saf. 2023. PMID: 37338799. doi:10.1007/s40264-023-01322-3

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。