MADE 1.0:从电子健康记录中提取药物、适应症与不良事件的首个自然语言处理挑战概述
MADE 1.0 挑战基于21例癌症患者的1089份纵向电子健康记录笔记,设计命名实体识别、关系识别及联合任务三项自然语言处理任务,11支队伍共提交41个系统。最佳系统在NER、RI和NER-RI任务上的F1分别为0.82、0.86和0.61,集成后提升至0.85、0.87和0.66。结果表明NER和RI进展显著,但联合任务仍有改进空间。
要点
- MADE 1.0提供专家标注的1089份电子健康记录笔记用于药物安全监测评测。
- 最佳系统在NER和RI任务的F1分别为0.82和0.86,联合任务仅0.61。
- 集成提交系统后三项任务F1分别提升至0.85、0.87和0.66。
核心结论
- 自然语言处理在临床领域NER和RI任务上取得显著进步,但NER-RI联合任务仍有较大改进空间。
研究设计与样本
- MADE 1.0挑战发布来自21名随机选取的癌症患者的1089份完全去标识化纵向电子健康记录笔记,由专家标注药物及不良事件信息。
- 挑战设计命名实体识别、关系识别及联合NER-RI三项共享任务,11支队伍共提交41个系统。
主要发现
- NER任务最佳系统F1为0.82,RI任务最佳F1为0.86,NER-RI任务最佳F1为0.61。
- 使用队伍提交的集成分类器后,三项任务F1分别提升至0.85、0.87和0.66。
关键数值与推荐
- NER任务识别药物及其属性(剂量、给药途径、持续时间、频率)、适应症、不良事件及严重程度。
- RI任务识别药物-适应症、药物-不良事件及属性关系。
- NER-RI任务评估联合执行NER和RI的模型。
临床怎么用
- 临床药物安全监测可参考MADE 1.0的标注体系和评测任务设计自然语言处理工具。
- 联合NER-RI任务性能较低,部署时需关注关系抽取的准确性。
局限性
- 数据集仅来自单一中心21名癌症患者的1089份笔记,外推性受限。
- 联合NER-RI任务F1仅0.61,集成后0.66,关系抽取性能仍需提升。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: This work describes the Medication and Adverse Drug Events from Electronic Health Records (MADE 1.0) corpus and provides an overview of the MADE 1.0 2018 challenge for extracting medication, indication, and adverse drug events (ADEs) from electronic health record (EHR) notes. OBJECTIVE: The goal of MADE is to provide a set of common evaluation tasks to assess the state of the art for natural language processing (NLP) systems applied to EHRs supporting drug safety surveillance and pharmacovigilance. We also provide benchmarks on the MADE dataset using the system submissions received in the MADE 2018 challenge. METHODS: The MADE 1.0 challenge has released an expert-annotated cohort of medication and ADE information comprising 1089 fully de-identified longitudinal EHR notes from 21 randomly selected patients with cancer at the University of Massachusetts Memorial Hospital. Using this cohort as a benchmark, the MADE 1.0 challenge designed three shared NLP tasks. The named entity recognition (NER) task identifies medications and their attributes (dosage, route, duration, and frequency), indications, ADEs, and severity. The relation identification (RI) task identifies relations between the named entities: medication-indication, medication-ADE, and attribute relations. The third shared task (NER-RI) evaluates NLP models that perform the NER and RI tasks jointly. In total, 11 teams from four countries participated in at least one of the three shared tasks, and 41 system submissions were received in total. RESULTS: The best systems F1 scores for NER, RI, and NER-RI were 0.82, 0.86, and 0.61, respectively. Ensemble classifiers using the team submissions improved the performance further, with an F1 score of 0.85, 0.87, and 0.66 for the three tasks, respectively. CONCLUSION: MADE results show that recent progress in NLP has led to remarkable improvements in NER and RI tasks for the clinical domain. However, some room for improvement remains, particularly in the NER-RI task.
引用
- Jagannatha A, Liu F, Liu W, et al. Overview of the First Natural Language Processing Challenge for Extracting Medication, Indication, and Adverse Drug Events from Electronic Health Record Notes (MADE 1.0). Drug Saf. 2019. PMID: 30649735. doi:10.1007/s40264-018-0762-z
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。