文献速递医学人工智能与信息学流行病学与生物统计

跨机构自然语言处理用于可靠的临床关联研究:一项方法学探索

Cross-institution natural language processing for reliable clinical association studies: a methodological exploration.
PubMed 文献 · J Clin Epidemiol · 2024-01-14

本研究在美国某学术健康系统的重症监护病房患者便利样本中开展多项关联研究,比较基于自然语言处理(NLP)提取与人工提取暴露变量的关联估计差异。研究覆盖不同NLP模型架构、训练范式、暴露变量、健康结局、缺失数据处理及测量误差校正方法,发现所有设定下两种提取方式的关联估计均存在实质性差异,且关联误差与模型总体F1分数仅弱相关。作者结论认为,使用NLP提取暴露变量估计的关联应谨慎解读,需进一步研究以设定NLP在医学关联研究中可靠使用的条件。

要点

  • 基于NLP提取暴露与人工提取暴露的关联估计在所有设定下均存在实质性差异。
  • 关联误差与NLP模型总体F1分数仅呈弱相关。
  • 作者认为NLP提取暴露的关联估计应谨慎解读,需进一步研究可靠使用条件。

核心结论

  • 使用NLP提取的暴露变量估计的关联应谨慎解读,尚需进一步研究以设定NLP在医学关联研究中可靠使用的条件。

研究设计与样本

  • 在美国某学术健康系统的重症监护病房患者便利样本中开展多项关联研究,比较基于NLP提取与人工提取暴露变量的关联估计。
  • 研究共纳入1,174名参与者(年龄中位数[四分位距]为61[50,73]岁;男性占60.6%)。
  • 另使用同一健康系统最多500份出院报告及外部健康系统2,528份报告训练NLP模型。
  • 关联研究在NLP模型架构(来自Transformers的双向编码器解码器、长短期记忆)、训练范式(训练新模型、微调现有外部模型)、提取的暴露变量(就业状况、居住状况、物质使用)、健康结局(是否签署不复苏/不插管医嘱、住院时长、院内死亡)、缺失数据处理(多重插补对比完整病例分析)及测量误差校正(通过回归校准)方面有所不同。

主要发现

  • 在所有设定下,基于NLP提取暴露与基于人工提取暴露的关联估计之间均存在实质性差异。
  • 关联误差与NLP模型的总体F1分数仅呈弱相关。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: Natural language processing (NLP) of clinical notes in electronic medical records is increasingly used to extract otherwise sparsely available patient characteristics, to assess their association with relevant health outcomes. Manual data curation is resource intensive and NLP methods make these studies more feasible. However, the methodology of using NLP methods reliably in clinical research is understudied. The objective of this study is to investigate how NLP models could be used to extract study variables (specifically exposures) to reliably conduct exposure-outcome association studies. STUDY DESIGN AND SETTING: In a convenience sample of patients admitted to the intensive care unit of a US academic health system, multiple association studies are conducted, comparing the association estimates based on NLP-extracted vs. manually extracted exposure variables. The association studies varied in NLP model architecture (Bidirectional Encoder Decoder from Transformers, Long Short-Term Memory), training paradigm (training a new model, fine-tuning an existing external model), extracted exposures (employment status, living status, and substance use), health outcomes (having a do-not-resuscitate/intubate code, length of stay, and in-hospital mortality), missing data handling (multiple imputation vs. complete case analysis), and the application of measurement error correction (via regression calibration). RESULTS: The study was conducted on 1,174 participants (median [interquartile range] age, 61 [50, 73] years; 60.6% male). Additionally, up to 500 discharge reports of participants from the same health system and 2,528 reports of participants from an external health system were used to train the NLP models. Substantial differences were found between the associations based on NLP-extracted and manually extracted exposures under all settings. The error in association was only weakly correlated with the overall F1 score of the NLP models. CONCLUSION: Associations estimated using NLP-extracted exposures should be interpreted with caution. Further research is needed to set conditions for reliable use of NLP in medical association studies.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Sushil M, Butte AJ, Schuit E, et al. Cross-institution natural language processing for reliable clinical association studies: a methodological exploration. J Clin Epidemiol. 2024. PMID: 38219811. doi:10.1016/j.jclinepi.2024.111258

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。