文献速递心力衰竭与心肌病

自然语言处理在多中心临床试验中判定心力衰竭:随机临床试验的二次分析

Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis of a Randomized Clinical Trial.
PubMed 文献 · JAMA Cardiol · 2024-01-01

本研究基于INVESTED随机临床试验,在157个中心、5260名心血管疾病患者中,外部验证了C3PO自然语言处理(NLP)模型对心力衰竭住院事件的判定效果,并与金标准临床事件委员会(CEC)判定进行比较。结果显示,C3PO模型与CEC判定具有良好一致性(原始一致率87%,κ 0.69),微调后模型一致率提升至93%,κ达0.82,接近人工判定的可重复性。

要点

  • C3PO NLP模型外部验证与CEC判定一致率87%,κ 0.69,敏感度94%、特异度84%。
  • 微调后模型与CEC一致率93%,κ 0.82,接近CEC评审者间可重复性(κ 0.85)。
  • NLP模型有望提升多中心临床试验事件判定的效率,但需进一步研究验证。

核心结论

  • 在一个医疗系统内开发的C3PO NLP模型,在外部队列多中心临床试验中判定心衰事件与金标准CEC具有良好一致性。
  • 对模型进行微调可提高一致性,并接近人工判定的可重复性。

研究设计与样本

  • 研究类型:对INVESTED随机临床试验的回顾性二次分析。
  • 研究对象:INVESTED试验纳入的5260名心血管疾病患者,来自美国和加拿大157个中心,于2016年9月至2019年1月期间比较两种流感疫苗。
  • 分析样本:共分析1973名患者的4060次住院记录;患者平均(SD)年龄66.4(13.2)岁,女性514例(27.4%),男性1432例(72.6%)。
  • 判定方法:各中心提交住院病历,由INVESTED中央CEC和C3PO NLP模型独立判定住院原因是否为心衰。
  • 模型训练:对C3PO模型进行微调(C3PO + INVESTED),并使用一半INVESTED住院数据训练一个新的NLP模型,在未用于训练的内部验证队列中评估。

主要发现

  • 在4060次住院中,CEC判定1074次(26%)为心衰。
  • C3PO NLP模型与CEC心衰判定原始一致率为87%(95% CI 86-88),κ为0.69(95% CI 0.66-0.72)。
  • C3PO NLP模型敏感度为94%(95% CI 92-95),特异度为84%(95% CI 83-85)。
  • 微调后的C3PO模型与CEC一致率为93%(95% CI 92-94),κ为0.82(95% CI 0.77-0.86)。
  • 新训练的NLP模型与CEC一致率为93%(95% CI 92-94),κ为0.83(95% CI 0.79-0.87)。

临床怎么用

  • NLP模型可在多中心临床试验中辅助心衰住院事件的自动判定,减少对人工CEC的依赖。
  • 微调后的NLP模型与CEC判定一致性高,接近人工可重复性,可能提高事件判定效率。

局限性

  • 研究为回顾性分析,结果需在前瞻性研究中验证。
  • NLP模型在特定试验中开发与验证,外推至其他试验或医疗系统需进一步评估。
  • NLP能否真正提升未来多中心临床试验效率尚需进一步研究。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

IMPORTANCE: The gold standard for outcome adjudication in clinical trials is medical record review by a physician clinical events committee (CEC), which requires substantial time and expertise. Automated adjudication of medical records by natural language processing (NLP) may offer a more resource-efficient alternative but this approach has not been validated in a multicenter setting. OBJECTIVE: To externally validate the Community Care Cohort Project (C3PO) NLP model for heart failure (HF) hospitalization adjudication, which was previously developed and tested within one health care system, compared to gold-standard CEC adjudication in a multicenter clinical trial. DESIGN, SETTING, AND PARTICIPANTS: This was a retrospective analysis of the Influenza Vaccine to Effectively Stop Cardio Thoracic Events and Decompensated Heart Failure (INVESTED) trial, which compared 2 influenza vaccines in 5260 participants with cardiovascular disease at 157 sites in the US and Canada between September 2016 and January 2019. Analysis was performed from November 2022 to October 2023. EXPOSURES: Individual sites submitted medical records for each hospitalization. The central INVESTED CEC and the C3PO NLP model independently adjudicated whether the cause of hospitalization was HF using the prepared hospitalization dossier. The C3PO NLP model was fine-tuned (C3PO + INVESTED) and a de novo NLP model was trained using half the INVESTED hospitalizations. MAIN OUTCOMES AND MEASURES: Concordance between the C3PO NLP model HF adjudication and the gold-standard INVESTED CEC adjudication was measured by raw agreement, κ, sensitivity, and specificity. The fine-tuned and de novo INVESTED NLP models were evaluated in an internal validation cohort not used for training. RESULTS: Among 4060 hospitalizations in 1973 patients (mean [SD] age, 66.4 [13.2] years; 514 [27.4%] female and 1432 [72.6%] male]), 1074 hospitalizations (26%) were adjudicated as HF by the CEC. There was good agreement between the C3PO NLP and CEC HF adjudications (raw agreement, 87% [95% CI, 86-88]; κ, 0.69 [95% CI, 0.66-0.72]). C3PO NLP model sensitivity was 94% (95% CI, 92-95) and specificity was 84% (95% CI, 83-85). The fine-tuned C3PO and de novo NLP models demonstrated agreement of 93% (95% CI, 92-94) and κ of 0.82 (95% CI, 0.77-0.86) and 0.83 (95% CI, 0.79-0.87), respectively, vs the CEC. CEC reviewer interrater reproducibility was 94% (95% CI, 93-95; κ, 0.85 [95% CI, 0.80-0.89]). CONCLUSIONS AND RELEVANCE: The C3PO NLP model developed within 1 health care system identified HF events with good agreement relative to the gold-standard CEC in an external multicenter clinical trial. Fine-tuning the model improved agreement and approximated human reproducibility. Further study is needed to determine whether NLP will improve the efficiency of future multicenter clinical trials by identifying clinical events at scale.

来源:JAMA Cardiol,查看原文。版权归原作者所有。

引用

  1. Cunningham, Singh, Reeder, et al. Natural Language Processing for Adjudication of Heart Failure in a Multicenter Clinical Trial: A Secondary Analysis of a Randomized Clinical Trial. JAMA Cardiol. 2024. PMID: 37950744. doi:10.1001/jamacardio.2023.4859

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。